Adversarially Robust Imitation Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Modern imitation learning (IL) utilizes deep neural networks (DNNs) asfunction approximators to mimic the policy of the expert demonstrations.
However,
DNNs can be easily fooled by subtle noise added to the input, which is evennon-detectable by humans.
This makes the learned agent vulnerable to attacks,especially in IL where agents can struggle to recover from the errors.
In such light,we propose a sound Adversarially Robust Imitation Learning (ARIL) method.
Inour setting, an agent and an adversary are trained alternatively.
The former withadversarially attacked input at each timestep mimics the behavior of an onlineexpert and the latter learns to add perturbations on the states by forcing the learnedagent to fail on choosing the right decisions.
We theoretically prove that ARILcan achieve adversarial robustness and evaluate ARIL on multiple benchmarksfrom DM Control Suite.
The result reveals that our method (ARIL) achieves betterrobustness compare with other imitation learning methods under both sensoryattack and physical attack.
Introduction
Imitation learning is a powerful and practical alternative to reinforcement learning for learningsequential decision-making policies.
Together with the computational advancement brought by deeplearning, imitation learning has shown great success in autonomous driving [1, 2], robot control [3],game AI [4] and motion capture [5].
Despite achieving human-level performance on many tasks, theexistence of adversarial examples [6] in DNNs and many successful attacks to sequential decisionmaking policies [7, 8, 9, 10] motivate us to study robust imitation algorithms.
The observations by the agent unavoidably contain uncertainty that naturally originates from sensorerrors or equipment inaccuracy.
A policy not robust to such uncertainty can lead to catastrophicfailures (e.g., a small piece of white scotch tape on a trafﬁc sign could cause an improper left turn,which will further lead to miles of unnecessary travel and even crush of the vehicle in the worst case).
Therefore, studying the fundamental principles of robust imitation learning is crucial.
One effective approach to induce robustness is domain randomization [11], a method where adesigner with expertise identiﬁes the components of the model that they are uncertain about.
A setof training environments is constructed where the uncertain components are randomized, ensuringthat the agent is robust on average to this set.
However, this requires careful parametrization of theuncertainty set as well as hand-designing of the environments.
A more easily automated approachis to formulate the problem as a zero-sum game and learn an adversary that perturbs the transitiondynamics [12, 13, 14].
If a Nash equilibrium of this problem is found, then that equilibrium providesa worst-case performance bound under the speciﬁed set of perturbations.
Speciﬁcally, we can searchover adversary policies f and agent policies π , so that min π ∈ Π max f ∈F J ( π, f ) , where J ( π, f ) isthe expected cumulative reward for the attacker policy.
Unfortunately, in most imitation learningsettings, quantitative reward from the environment is unavailable.
In this work, we propose a theoretically grounded framework of adversarially robust imitation learning(ARIL), where we build a surrogate loss to measure the distance between the actions generated fromπ and the actions from the expert.
We consider two ways of attack: sensory attack and physicalattack.
In the sensory attack setting, the adversary only modiﬁes the inputs to π (e.g., the underlying true state of the environment is untouched as s t ), while in the physical attack setting, the adversarydirectly modiﬁes the actual physical state, i.e., it changes the underlying state to s (cid:48) t .
We considerattacking under interactive imitation learning (IL) setting where the expert is available for queries atany time step [15].
We evaluate our method in DM Control Suite [16], (e.g. HalfCheetah, Swimmer,
Ant), under both sensory attack and physical attack settings.
Our method signiﬁcantly improves therobustness under adversarial attacks over the baseline methods.
To summarize, this paper makes four contributions: (1) We study the theory and practice of Adversarially Robust Imitation Learning (ARIL).
(2) We systematically deﬁne sensory attack and physicalattack.
(3) We prove that ARIL can achieve the adversarial robustness under realizability assumption, robust expert assumption, and optimal adversary assumption.
(4) We show the efﬁciency ofour method in both attack stage and defense stage on multiple DM Control Suite benchmarks incomparison with other adversarial attack methods and other robust imitation learning methods.
Related Work
Adversarial Attack Adversarial attacks can be divided into white-box attacks and black-boxattacks.
In white-box attacks, the attack has access to the structure, algorithm, and system parametersof the target model.
In black-box attacks, the attack cannot obtain information from the target model.
Instead, it can only interact with the target model by querying with the input and observing the output.
In our setting, we consider white-box attacks.
Szegedy et al.
[17] ﬁrst pointed out the intriguingproperty of neural networks and proposed a method called Large-BFGS to ﬁnd adversarial samples.
Goodfellow et al.
[18] proposed a Fast Gradient Sign method (FGSM) to generate small perturbationsto fool the neural networks by calculating the gradient of the loss function concerning the inputs.
Kurakin et al.
[19] proposed Targeted FGSM to generate adversarial samples towards a target classand Iterative FGSM to apply FGSM multiple steps with small step sizes.
Tramer et al.
[20] proposed arandomized single-step attack to escape the non-smooth vicinity of the data point.
Kurakin et al.
[19]apply adversarial training on large-scale datasets.
Huang et al.
[7] showed that adversarial examplesgenerated from FGSM could degrade test-time performance of a well-trained NN policy in RL.
Sun et al.
[21] studied the safe sequential decision-making problem under the setting of adversarialcontextual bandits which is a one-step RL problem.
Lin et al.
[9] considered strategically-timedattack where the attacker picks speciﬁc time steps to attack.
Han et al.
[22] considered attackingan RL algorithm during training time.
All these previous works mainly focus on applying attacksdeveloped from the supervised learning literature to RL setting, while in this work, we focus onlearning adversarial attacks in imitation learning.
Robust Imitation Learning Imitation learning aims to learn sequential decision-making policiesfrom expert demonstrations.
However, the agent can be vulnerable to cascading failures when thetrajectory diverges from the demonstrations [23].
Laskey et al.
[24] proposes to inject noise into thesupervisor’s demonstration policies, which forces the supervisor to demonstrate how to recover fromerrors.
Wang et al.
[25] utilizes VAEs [26] and GAIL [27] to learn semantic policy embeddings inimitation learning, which learns robust diverse gaits with limited demonstrations.
For robust imitationlearning with different-quality demonstrations, Tangkaratt et al.
[28] makes use of a probabilisticgraphical model to estimate the quality of demonstrations.
Brown et al.
[29] proposes a safe androbust Bayesian reward learning through preferences over imitating demonstrations.
Shin et al.
[30]performs the derivative-free random search optimization method with linear policies.
In our work,we focus on the robust imitation learning against adversary attacks, which can cause catastrophicfailure to DNN-based agents with trivial perturbations.
Preliminaries
Before we delve into the details of ARIL, we ﬁrst outline our terminology.
In the rest of the paper, wefocus on ﬁnite-horizon Markov Decision Process (MDP), which is deﬁned as ( S , A , H, P, r ) where
S is the state space, A is the action space, r is a reward function, P is the transition and H ∈ N + isthe ﬁnite horizon.
A policy π : S → ∆( A ) is a mapping that maps from states to distributions overaction space, i.e., π ( a | s ) is the probability of picking a given s .
3.1 Sensory Attack vs Physical Attack
For any pair of policy and attack ( π, f ) , we can treat the composition of them as a policy: π ◦ f :
S → A , i.e., a ∼ ( π ◦ f )( s ) is equivalent to ﬁrst generate stochastic adversarial example s (cid:48) ∼ f ( ·| s ) ,and then sample action accordingly a ∼ π ( ·| s (cid:48) ) .
Denote d π as the average state distribution inducedby policy π .
As π ◦ f can be understood as a policy, d π ◦ f will be the state distribution induced byπ ◦ f .
We ﬁrst introduce the sensory attack setting where the adversary can only modify the sensory inputs of the RL agent, but not the underlying physical state.
Denote a trajectory asτ = { s 1 , s (cid:48) 1 , a 1 , ..., s H , s (cid:48)
H , a H } generated from the composite policy π ◦ f , the likelihood ofsuch trajectory is ρ π ◦ f :
[Equation]
[Equation]
where we denote P ( s 1 )
=
P ( s 1 | s 0 , a 0 ) as the initial state distribution.
Note that here adversarymodiﬁes state s t to s (cid:48) t and then the policy π only sees s (cid:48) t .
For physical attack setting, the adversary will have the ability to modify the underlying physical states.
Again, we denote a trajectory as τ = { s 1 , s (cid:48) 1 , a 1 , ..., s H , s (cid:48) H , a H } generated from the compositepolicy π ◦ f , the likelihood of such trajectory now is:
[Equation]
Note the difference between Eq. 2 and Eq. 1, where transition probability is conditioned on attackedstate.
In physical attack (Eq. 2), the adversary modiﬁes the state s to s (cid:48) , and then the environmentactually generates the next state based on s (cid:48) , as the underlying physical state is now s (cid:48) .
The sensoryattack is similar to attacks considered in today’s supervised learning setting, where the physical attackis similar to the robust control setting (e.g., a min-max formulation of H ∞ optimal control [31] underphysical disturbances).
Note that under both settings, the attacks are sequential: the adversary can affect the learner’s decisionvia corrupting the state s , which will, in turn, affect the future states.
In supervised learning, an attackon a given sample will not affect the future samples, since samples are i.i.d in supervised learning.
3.2 Expert Robustness
For the expert π e , we have:
[Equation]
In practice consider the setting where the expert π e is a human driver who will not be affected byadversarial examples with small perturbations (e.g., human can easily tell a stop sign even with asticker on it).
We later will relax the notion of expert robustness to include the possibility that theexpert will be affected by adversarial attacks.
We consider interactive imitation learning setting proposed in [32] where the expert π e is alwaysavailable during training time.
A learner can query the expert at any state s during training to getfeedback π e ( s ) .
Adversarially Robust Imitation Learning
The goal of adversarial robust imitation learning (ARIL) is to learn a policy that can achieve goodperformance under all possible attacks:
[Equation]
where J ( π, f ) is the expected total cost of executing policy π with examples corrupted by attack f :
[Equation]
We consider a loss function (cid:96) ( a, a (cid:48) ) that measures the difference between two actions a and a (cid:48) .
Forinstance, (cid:96) ( a, a (cid:48) ) could be zero-one loss if A is discrete, or l2 distance between two actions if A iscontinuous.
If we can control the long-term prediction error of the learned policy ˆ π under the worst possibleadversary,
[Equation]
then we can upper bound the performance of ˆ π at test time with the existence of an arbitrary adversaryf ∈ F .
To see that, let us consider the setting where A is discrete and (cid:96) ( a, a (cid:48) ) is the zero-one loss:
[Equation]
Here, Q e denotes the state-action value of the expert policy and Q max denotes the upper bound of
Q e .
Namely, if δ is small, the learned policy ˆ π can achieve similar performance as the expert, evenunder an adversarial attack f during the execution of ˆ π .
This justiﬁes the reason for ﬁnding ˆ π thatminimizes the objective function in Eq.
Let us deﬁne the objective as follows:
[Equation]
[Equation]
We perform an iteratively alternating update for π and f .
4.1 Update π with ﬁxed f
At iteration t , given the current f t , and π t , we want to update π t to π t +1 .
We consider the lossfunction with respect to π at round t :
[Equation] [Equation]
which is a classiﬁcation loss where features are corrupted sampled s ∼ ˜
d π t ◦ f t , and labels arecomputed from expert query π e ( s ) .
Note that for discrete action space, the above loss function is simply a classiﬁcation loss of theclassiﬁer π , with features generated from the distribution ˜ d
π t ◦ f t , and the ground truth label isgenerated from π e .
We perform no-regret update on (cid:96) t ( π ) to generate π t +1 .
[Equation]
[Equation] Note that the above optimization can be understood as a classiﬁcation task where the features aresampled from the mixture of distributions { ˜ d π i ◦ f i } ti =1 with equal weight 1 /t , and the correspondinglabels are generated based on π e ◦ f i for a data point sampled from ˜ d
π i ◦ f i , ∀ i ∈
[ t ] .
In practice,we need to use samples to replace expectations, which leads us to a Data Aggregation style update(DAgger [32]) for π .
4.2 Update f with ﬁxed π
At iteration t , given the current f t and π t , we want to update f t to f t +1 .
We consider the objectivefunction with respect to f at round t as:
[Equation]
We want to ﬁnd f that maximize the above objective function, namely, increasing the classiﬁcationloss of the current policy π t as much as possible.
Note that different from the loss function forupdating π , here the expectation is deﬁned with respect to the ﬁxed π t and f .
If π t is a stochastic
1: Input: MDP M , policy class Π , expert policy π ∗ , set of adversaries F , a supervised learning
2: Initialize a dataset D = ∅ 3: Initialize a policy π 0 ∈ Π 4: for t = 0 to N do 5: f t = arg max f ∈F E s ∼ d πt ◦ f
[ (cid:96) ( π t ( f ( s )) , π e ( s ))] (RL Step) 6:
Execute π t ◦ f t to generate a set of non-corrupted states { s i } ni = 7:
Query expert during execution to get labels { π e ( s i ) } ni = 8:
Data Aggregation: D
= D + { f t ( s i ) , π e
( s i ) } ni = 9: π t +1 =
L EARNER ( D ) (Supervised Learning, e.g., Classiﬁcation if A is discrete ) 10: end for policy, and f is parameterized as f ( s ; θ ) , then one can use policy gradient technique to compute the∇ θ ˜ (cid:96) t ( f ) as follows:
[Equation]
(cid:48)
[Equation]
(cid:96) (
[Equation]
where the second step simply uses chain rule, and in the last step we simply treat π ◦ f together asa policy that maps from states to actions—hence the Q function is deﬁned with respect to π t ◦ f t .
Intuitively, we want to optimize f θ such that it increases the likelihood of generating actions viaπ t to increase the total “reward"—the long-term prediction error (with respect to π e ) of the policyπ t .
Using the above gradient to perform gradient ascent to optimizing f θ is similar to classic policygradient methods and in practice, one can use existing techniques developed for PG here as well (e.g.,natural gradient, actor-critic where the critic is Q π t ◦ f t ).
4.3 Algorithm
We summarize the algorithm ARIL in Alg. 1, where every round, we update the adversary f and thepolicy π .
Given a policy π t , we aim to ﬁnd the adversary f t from F that increases the long-termprediction error of π t (w/ expert’s prediction) as much as possible.
Once we ﬁnd such adversaryf t , we generate data by executing the composite policy π t ◦ f t on the real system to generate states{ s i } ni =1 ∼ d π t ◦ f t and labels { π e ( s i ) } ni =1 (this procedure requires interactively querying expert, justas DAgger does [32]).
We then corrupt the data by applying the latest adversary f t at every data points t , and appendix the new dataset { f t ( s i ) , π e ( s i ) } ni =1 to the aggregated dataset D .
Lastly, we trainthe new policy using the black box supervised learner on the aggregated dataset D .
The data aggregation technique (DAgger [32]) is used to ensure stability in the process of updating πand corresponds to the speciﬁc no-regret learner FTL as we explained before.
However, any otherno-regret learner can be used here as well, such as Online Gradient Descent [33], with the potentialbeneﬁt of no need to maintain an ever-growing dataset D .
In theory, optimizing f t is challenging as the objective function could potentially be non-convex here.
However, in practice for parameterized adversary f , as we showed in the previous section, we cancompute the gradient of f using classic policy gradient technique, which in turn allows us to leveragepractically effective techniques such as natural gradient, actor-critic, to optimize f t .
Note that theoptimization procedure for f t does not require any expert data.
4.4 Analysis
We consider the following setting.
First, for optimization in f t , we assume we can achieve nearoptimality:
[Equation] [Equation]
with (cid:15) rl ∈ R + .
Namely, we do not assume we can ﬁnd the global optimal adversary at every round.
The (cid:15) here quantiﬁes the performance of the potential RL algorithm in terms of optimizing ˜ (cid:96) t ( f ) withrespect to f .
For simplicity, we consider the realizability setting : there exists a policy π ∈ Π , such that for anystate s ∈ S , E a ∼ π ( s )
[ a (cid:54) =
π e ( s )] = 0 .
Namely there exists at least one policy in policy class Π thatcan predict expert’s action at any state s ∈ S .
Relaxing the realizability setting to agnostic setting isstraightforward here.
Theorem 4.1.
As T → ∞ , Alg.
1 outputs a policy π i [Equation] ∈ { π t } Tt =
such that:
[Equation]
The above theorem indicates that (1) under the realizability assumption (i.e., policy class Π is richenough to approximate π e well), (2) the assumption that the expert is robust with respect to anyattack (i.e., (cid:15) e is deﬁned as the error between π e ( s ) and π e ( f ( s )) , which is negligible), and (3) theassumption that we can near-optimally solve the RL-like optimization problem
max f ∈F ˜ (cid:96) t ( f ) atevery round t (i.e., (cid:15) rl is small), then Alg.
1 can achieve adversarial robustness.
Please refer tosupplementary material for proof.
Experiments
We evaluate our proposed adversary and robust imitation algorithm with baseline methods using DM
Control Suite [16].
We compare our method with FGSM [18].
Considering FGSM is commonlyused in supervised learning setting, we apply FGSM to modify state s h such that it hurts the currentone-step prediction of π t , i.e. f fgsm ( s h ) =
s h + (cid:15) sign ( ∇ s E a ∼ π t ( s ) ( (cid:96) ( a, π e
( s h )))
| s h ) .
Our algorithm achieves better robustness under both sensory attack and physical attack.
The detailsof the datasets and other experimental settings are described below.
5.1 ARIL with Sensory Attack
In sensory attack, the adversary only modiﬁes the sensory inputs of the agent, but not the underlyingphysical state, which can be treated as sensory noise in real scenarios.
Environment Setting We investigate the performance of attacks on three continuous control tasksfrom OpenAI gym robotic benchmarks [34]: Ant-v2 , HalfCheetah-v2 and Swimmer-v2 .
For theconvenience of training and testing, we set the horizon H to 1000 in all the benchmark environments.
We only attack the zero-order state in the robotics environments, i.e. the positions and angles otherthan velocities (ﬁrst-order state).
The attack is bounded by ± (cid:15) ( (cid:15) is a small positive number), whichvaries according to the robustness of each environment.
The (cid:15) is chosen to have no more than40% performance drop with FGSM attack and its values for all experimental setups can be foundin supplementary materials.
For comparison, all attack methods are bounded to the same (cid:15) whenapplying to the same robotics environment.
Training Details Our implementation of ARIL is built on top of rllab [35] and uses Trust Region
Policy Optimization (TRPO) [36].
Throughout the experiments, both policy π and adversary f θ areparameterized with two hidden layers.
The output layer of adversary uses tanh as activation functionand then multiples (cid:15) to restrict the output in [ − (cid:15), (cid:15) ] .
The standard deviation of f θ is initialized to 0 .
1 (cid:15)and the attack to state s is sampled from N ( f θ ( s ) , 0 .
1 (cid:15) ) .
We also clip the sampled adversarial noiseto [ − (cid:15), (cid:15) ] .
Hyperparameters are selected by grid search and the values for all experimental setups canbe found in supplementary materials.
Results We ﬁrst compare our adversary with the baseline Fast Gradient Sign Method (FGSM) [18]
In both our method and FGSM, the perturbation added to each state is bounded to a small value suchthat it’s imperceptible as shown in the supplementary material.
From Figure 1 (top), we ﬁrst observe that FGSM attack in imitation learning works well.
By forcingthe policy to generate actions that are far away from the expert’s, FGSM attack is effective andsuccessfully decreases the performance of π .
The performance of FGSM and our attacks in allsettings can be found in Table 1 (Adversary).
In all tasks, our attack outperforms the FGSM attack.
We experimentally demonstrate that through maximizing the total difference to expert policy, theadversarial attack could signiﬁcantly decrease the total reward.
We also show that naively applyingadversarial attack techniques developed for supervised learning to a sequential decision-making agentis not effective since the attack is myopic.
This result proves the efﬁciency of our attacks even withoutever accessing to reward signals.
We then compare ARIL with another robust imitation learning method, DART [24].
DART learns toinject an optimized level of noise into the expert demonstrations in imitation learning, which forcesthe expert to demonstrate how to recover from the errors.
A noise distribution, p ( ξ | π θ ∗ , ψ ) , is injectedto the demonstration distribution π θ ∗ ( u | x, ψ ) , where ψ denotes the sufﬁcient statistics that deﬁne thenoise distribution.
In our experiments, we set π θ ∗ ( u | x, ψ ) as a Gaussian distribution, N ( π θ ∗ ( x ) , Σ) .
The optimal parameter ˆ ψ is estimated by the anticipated ﬁnal robot’s error and subsequently scalingthe current simulated error to this level, denoted as E p ( ξ | π θ ∗ , ˆ ψ ) (cid:80) t (cid:96) ( u t , π π ∗ ( x t )) .
For quantitativecomparison, we train an adversary to attack DART using our proposed attacker (ARIL-A).
As shownin Table 1 (bottom), our method signiﬁcantly improves the robustness over DAgger [32] (3rd row and5th row).
ARIL also achieves better robustness compared with DART (4th row and 5th row) in alltasks under the worst possible adversary.
[Table]
5.2 ARIL with Physical Attack
In physical attack
[Table]
the attacker can also affect the underlying transition dynamics.
Considering theattacker changes the actual state of in the environment, the transition model can be described as s t +1 =
f ( P ( ·| s t , a t )) .
Namely, the actual zero-order states are changed under the attack.
We cantreat physical attack as an external perturbation (e.g. a kick on the robot or an unpredicted bump onthe road).
Environment Setting Similar to sensory attack, the attack is bounded by ± (cid:15) .
It’s worth mentioningthat (cid:15) is different for sensory attack and physical attack even in the same environment.
Since physicalattack with the ability to change underlying dynamics is more effective than sensory attack.
The (cid:15)are chosen to have no more than 40% performance drop with FGSM attack and its values for allexperimental setups can be found in supplementary materials.
Results We ﬁrst compare our adversary with the FGSM baseline.
As one might expect, the physical attack is more effective than the sensor attack.
Under the physical attack setting, π is more likely tofail.
Our experiments support this viewpoint in the sense that (cid:15) for sensory attack needs to be largerthan the one for the physical attack to achieve similar attack performance.
Again, in all tasks, ourattack outperforms the FGSM attack.
We also compare ARIL with the DART (Table 2).
This resultproves the efﬁciency of our robust imitation algorithm even the underlying transition dynamics arechanged.
[Table]
Conclusion
In conclusion, we propose an adversarially robust imitation learning algorithm.
We theoretically provethat under realizability assumption, robust expert assumption, and optimal adversary assumption,
ARIL achieves adversarial robustness.
We evaluate our method on a set of continuous control tasksfrom OpenAI Gym and show that our method achieve better robustness under both sensory attack andphysical attack compare with baselines.
We hope that our works can inspire more studies in robustimitation learning methods when the quantitative reward from the environment is unavailable.