P REVENTING I MITATION L EARNING WITH A DVERSARIAL P OLICY E NSEMBLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Imitation learning can reproduce policies by observing experts, which poses a problem regarding policy privacy.
Policies, such as human, or policies on deployed robots, can all be cloned without consent from the owners.
How can we protect our proprietary policies from cloning by an external observer?
To answer this question we introduce a new reinforcement learning framework, where we train an ensemble of near-optimal policies, whose demonstrations are guaranteed to be useless for an external observer.
We formulate this idea by a constrained optimization problem, where the objective is to improve proprietary policies, and at the same time deteriorate the virtual policy of an eventual external observer.
We design a tractable algorithm to solve this new optimization problem by modifying the standard policy gradient algorithm.
Our formulation can be interpreted in lenses of conﬁdentiality and adversarial behaviour, which enables a broader perspective of this work.
We demonstrate the existence of “non-clonable” ensembles, providing a solution to the above optimization problem, which is calculated by our modiﬁed policy gradient algorithm.
To our knowledge, this is the ﬁrst work regarding the protection of policies in Reinforcement Learning.
I NTRODUCTION
Imitation learning and behavioral cloning provide really strong ability to create powerful policies, as seen in robotic tasks (Laskey et al., 2017; Finn et al., 2017; Codevilla et al., 2019; 2017; Pomerleau, 1988; Bojarski et al., 2016).
Other ﬁelds have developed methods to ensure privacy (Al-Rubaie & Chang, 2019; Papernot et al., 2016), however, such work do not offer protection against policy cloning.
In this work, we tackle the issue of protecting policies by training policies that aim to prevent an external observer from using behaviour cloning.
Our approach draws inspiration from imitating human experts, who can near-optimally accomplish given tasks.
The setting which we analyze is presented in Figure 1.
We wish to ﬁnd a collection of experts, which as an ensemble can perform a given task well, however, also targets behaviour cloning through adversarial behaviour.
Another interpretation is that this collection of experts represents the worst case scenario for behaviour cloning on how to perform a task “good enough”.
Imitation learning frameworks generally make certain assumptions of the optimality of the demonstrations (Ziebart et al., 2008; Levine, 2018), yet never considered the scenario when the experts speciﬁcally attempt to be adversarial to the imitator.
We pose the novel question regarding this assumption: does there exist a set of experts that are adversarial to an external observer trying to behaviour clone?
We propose Adversarial Policy Ensembles (APE), a method that simultaneously optimizes the performance of the ensemble and minimizes the performance of policies eventually obtained from cloning it.
Our experiments show that APE do not suffer much performance loss from an optimal policy, while causing, on average, the cloned policy to experience over 5 times degradation compared to the optimal policy.
Our main contributions can be summarized as follows:
• We introduce a novel method APE, as well as the mathematical justiﬁcation of the notion of adversarial experts.
• By modifying Policy Gradient (Sutton et al., 2000), a common reinforcement learning algorithm, we suggest a tractable scheme for ﬁnding an optimal solution for this objective.
• We demonstrate the solution by numerical simulations, where we show that a cloned policy is crippled even after collecting a signiﬁcantly large number of samples from a policy ensemble.
To our knowledge, not only is this the ﬁrst work regarding the protection of policies in reinforcement learning, but it is also the ﬁrst to represent adversarial experts.
P RELIMINARIES
We develop APE in the standard framework of Reinforcement Learning (RL).
The main components we use are Markov Decision Processes, Policy Gradient (Sutton et al., 2000), policy ensembles, and behaviour cloning, which we review below.
2. M ARKOV D ECISION P ROCESS
A discrete-time ﬁnite-horizon discounted Markov decision process (MDP) M is deﬁned by ( S , A , r, p, p 0 , γ, T ) where S is the state space, A is the action space, r : S ×
A → R is the reward function, p ( s t +1 | s t , a t ) is the transition probability distribution, p 0 :
S
→ R + is the initial state distribution, γ ∈ (0 , 1) is the discount factor, and T is the time horizon.
A trajectory τ ∼ ρ π , sampled from p and a policy π : S ×
A → R + , is deﬁned to be the states and actions tuple ( s 0 , a 0 , ...
s T − 1 , a T − 1 , s T ) , whose distribution is characterized by ρ π .
Deﬁne the return of a trajectory to be r ( τ ) =
(cid:80) T − 1 t =
γ t r ( s t , a t ) to be the sum of discounted rewards seen along the trajectory, and deﬁne a value function V π : S → R to be expected return of a trajectory starting from state s , under the policy π .
The goal of reinforcement learning is to ﬁnd a policy that maximizes the expected return E τ ∼ ρ π [ r ( τ )] .
2. P OLICY G RADIENT
Policy Gradient (PG)
(Sutton et al., 2000) aim to directly learn the optimal policy π , parameterized by θ , by repeatedly estimating the gradient of the expected return, in one of many forms, shown in Schulman et al.
In our work, we follow notation similar to that of (Schulman et al., 2015; 2017) and estimate ∇ θ E τ ∼ ρ π [[-(cid:80) Tt =
-] r ( τ )] using the advantage, which is estimated from a trajectory τ , A πτ ( t ) =
R τ ( t ) − V π ( s t ) , where R τ ( t ) =
(cid:80) T − 1 t (cid:48) =
t γ t (cid:48) r ( s t (cid:48) , a t (cid:48) ) is the sum of the reward following action a t .
Here, the value function is learned simultaneously with the the policy, and so the advantage will use ˆ
V π as an estimate for V π .
2. P OLICY E NSEMBLE (PE)
A Policy Ensemble (PE) is similar to the notion of contexts and skills (Achiam et al., 2018; Eysenbach et al., 2018; Sharma et al., 2019) which we discuss in Section 3. We denote a PE by π c , where each π c ( i ) , i ∈ { 1 , 2 , ...n } represents an expert.
To rollout the PE, an expert is chosen at random (in our case uniform), and the expert completes a trajectory.
Each expert policy π c ( i ) ( a | s ) can be viewed as a policy conditioned on a latent variable c , π ( a | s, c ) .
Although π c consists of multiple policies, it is important to note that it itself is still a policy.
2. B EHAVIOUR C LONING
To behaviour clone an expert policy (Widrow & W. Smith, 1964), a dataset of trajectories D consisting of state action pairs ( s, a ) are collected from the the expert rollouts.
Then, a policy parametized by φ is trained to maximize the likelihood of an action given a state, (cid:80) ( s,a ) ∈D − log π φ ( a | s ) .
When cloning π c , D will not contain information of the latent variable c , and so the cloned policy will marginalize it out.
Thus, the observer will clone:
[Equation]
We stress that this policy does not exist until π c is behaviour cloned.
π o is a ﬁctitious policy to represent what would happen in the best case scenario of the observer having access to inﬁnite data from π c to clone into π o .
The scope of this paper is to speciﬁcally prevent behavioral cloning from succeeding.
Other imitation learning approaches such as inverse reinforcement learning (Abbeel & Ng, 2004; Ng & Russell, 2000;
Levine et al., 2011) and adversarial imitation learning (Ho & Ermon, 2016; Peng et al., 2018) require rollouts of non-expert policies in the environment, which may be costly, and thus are not considered.  R ELATED W ORK
Adversarial Attacks in RL: Our notion of adversarial policies is inextricably related to other adversarial methods that target RL such as Lin et al.
(2017); Behzadan & Munir (2017), that add adversarial perturbations to policy input. Other adversarial attacks include poisoning the batch of data used when training RL (Ma et al., 2019), and exploitation in the multi-agent setting (Gleave et al., 2019). However, these methods all present as active attacks for various learning techniques. Our method, instead, passively protects against cloning.
Privacy in RL: With regards to protection, our work is related to differential privacy (Al-Rubaie &
Chang, 2019). Differential privacy in RL can be used to create private Q-functions (Wang & Hegde, 2019) or private policies (Balle et al., 2016), which have private reward functions or private policy evaluation. However, we would like to emphasize that our motivation is to prevent cloning, and thus protecting the policies, rather than protecting a dataset. In fact, we make the assumption that the observer can perform behaviour cloning on as much data as desired.
Imitation Learning: Since we comply to the standard imitation learning setting of cloning from a dataset with many experts providing the demonstrations, latent variables w.r.t.
imitation learning is well-studied.
For example, Codevilla et al. (2017) show that conditioning on context representation can make imitation learning a viable option for autonomous driving. Li et al. (2017) demonstrate that the latent contextual information in expert trajectories is often semantically meaningful.
As well,
Providing extra context variables to condition on also appears in forms of extra queries or providing labels (Brown et al., 2019; de Haan et al., 2019; Hristov et al., 2018).
Our method is different as instead of experimenting for success in imitation learning, we study how to prevent it.
Multiple Policies: (Achiam et al., 2018;
Eysenbach et al., 2018; Sharma et al., 2019) have similar schemes of sampling a latent variable and ﬁxing it throughout a trajectory, although their latent variables (contexts or skills) are used to solve semantically different tasks.
The reason to solve different tasks is due to the objective of using the context variable/skills for learning in an unsupervised setting.
Our approach differs in both motivation and implementation, as we learn experts that all solve the same task, and constrain so that observers can not clone the policy.
A PE π c can also be viewed as a mixture of experts (Jacobs et al., 1991), except the gating network assigns probability 1 to the same expert for an entire trajectory.
As such, we do not learn the gating network, although it may still be useful to see π c as a special case of a mixture of experts where the gating network learns immediately to ﬁx the expert for each trajectory.
There are also methods such as OptionGAN (Henderson et al., 2018), which uses a mixture of experts model to learn multiple policies as options with access to only expert states.
Zhang et al.
(2019) also proposes a method to train multiple policies that complete the same task but uses the uncertainty of an autoencoder as a reward augment.
Their motivation is to ﬁnd multiple novel policies, while our motivation has no connection to novelty.
Due to these differences in motivation, they train each policy one after the other, while our policies are trained simultaneously.
Policy ensembles are also used in the multi-task and goal conditioned settings in which case the task that is meant to be solved can be viewed as the context.
Marginalizing out the context variable (Equation 1) of these context-conditioned policies is studied in the case of introducing a KL divergence regularizing term for learning new tasks (Goyal et al., 2019) and for sharing/hiding goals (Strouse et al., 2018).
However, the main motivation is different in that both Goyal et al.
(2019); Strouse et al.
(2018) use π o to optimize mutual information, while we directly optimize its performance.
M ETHOD 4. O BJECTIVE
We wish to have experts that can perform the task, while minimizing the possible returns of the cloned policy, denoted in Equation 1.
We modify the standard RL objective to be:
[Equation]
where α is a parameter that lower bounds the reward of the policy ensemble.
This translates to maximizing the unconstrained Lagrangian:
[Equation]
where 1 /β is the corresponding Lagrangian multiplier, and is subsumed into the returns collected by the policy ensemble.
We refer to PE that optimizes this objective as Adversarial Policy Ensembles (APE).
There is a natural interpretation of the objective in Equation 2.
Human experts tend to be “good enough”, which is reﬂected in the constraint.
The minimization is simply ﬁnding the most adversarial experts.
Although we assume that the observer can only map states to actions, it may be the case that they can train a sequential policy, which is dependent on its previous states and actions.
Our method can be generalized to sequential policies as well, and the impact of such observers is discussed in the
Section 6.
4. M ODIFIED P OLICY G RADIENT A LGORITHM
Intuitively, since there are the returns of two policies that are being optimized, both should be sampled from to estimate the returns.
The optimization procedure, as well as gradient back propagation, is summarized in Figure ?? .
We show how we can modify PG to train APE, by maximizing Equation 3.
The two terms suggest a simple scheme to estimate the returns of the policy ensemble twice: once using π c that we wish to maximize, and a second time using π o , which approximates the returns of an eventual observer who tries to clone the policy ensemble.
Along with our PE, we train value functions ˜
V π c ( i ) for each expert, jointly parameterized by φ which estimates V π c ( i ) − βV π o .
The loss function for the value functions of two sampled trajectories τ 1 , τ 2 is
[Equation]
The policy gradient update from N 1 and N 2 trajectories is then
[Equation]
where
[Equation]
[Equation]
where c ( i ) identiﬁes the chosen expert of the trajectory. , and ˜ A π c ( i ) τ( t ) = R τ 1 ( t ) − ˜ V
[Equation]
( s
[Equation]
and
˜
A π o τ 2 ( t ) = − βR τ 2 ( t ) − ˜ V
π o ( s t ) are the modiﬁed advantage functions. The − β that is in the advantage in G 2 optimizes against the performance of the observed policy π o .
The gradient G 1 for π c is straightforward. However, to estimate the gradient G 2 for π o which is an ﬁctitious policy, we sample from it by ﬁrst re-sampling the context of the expert at each state, and then sampling an action from the context. The back-propagation occurs to π c ( i ) ( a | s ) for the context sampled at each state.
Practical implementation details can be found in A.2.
The intuition is as follow. While sampling π o , if a selected action causes high return, we should decrease the probability, which lowers the expected reward of π o .
Combined, the two gradients will cause the PE to select actions that achieves have high reward, and are detrimental to the observer.
Equations 4 and 5 formulate our PG approach of APE, which is summarized in Algorithm 1.
Require: θ , φ , M , β 1: for each iteration do : 2:
Generate trajectories τ 1 with π c from M for Equation3:
Generate trajectories τ 2 with π o from M for Equation4:
Calculate Equation 5 to perform a gradient update on the PE θ ← θ + α θ ˆ ∇ θ J τ 1 ,τ 2 ( θ ) 5:
Update the value function φ ← φ − α φ ˆ ∇ φ J τ 1 ,τ 2 ( φ ) as determined by Equation 4. 6: end for
E XPERIMENTS
We perform experiments on a navigation task, where the objective is to reach a goal state as fast as possible.
The purpose is to illustrate that an APE can cause the cloned policy to take signiﬁcantly longer to reach the goal state.
We do so by ﬁrst training a PE and behaviour cloning it.
We then compare the performance of the PE to that of the clone.
We use a discrete environment to best demonstrate the validity of the equation.
This is because all discrete policies can be parameterized, which is not true in continuous, where typically Gaussian parameterization is used.
As such, continuous environments would have to make assumptions about how both the PE and the cloner parameterizes policies, as well as tackle problems of distributional drift, which we would like to avoid.
However, with these assumptions, our setting can extend to the continuous domain.
In our experiments, we use a 10 × 10 grid-world environment as our main testbed.
This is to have large enough expression that would not be found in smaller grids, while still small enough to visualize the behaviour of the APE.
The discrete actions will show precisely how the experts can be jointly adversarial.
Using gridworld allows for precise expected return estimates.
In an environment where closed-form returns cannot be calculated, approximation error can accumulate through estimating the returns of both the trained PE and the clone.
This noise would only increase in continuous state space, where the returns of π o may not be tractable to estimate due to issues such as distributional drift (Ross et al., 2010; Codevilla et al., 2019; de Haan et al., 2019).
Our results answer the following questions.
How much optimality is compromised?
How useless can we make the cloned policy?
Is it possible to use non APE to prevent behaviour cloning?
5. T RAINING
Even though our method can compute a policy ensemble with any ﬁnite number of experts, we chose to visualize a solution with 2 experts, which is sufﬁcient to reveal the essential properties of the method.
Speciﬁcally, we train n = 2 tabular experts with PG-APE.
Our code is written in Tensorﬂow (Abadi et al., 2016), and will be publicly available on GitHub.
Training details and hyper-parameters are in Section A.1 of the Appendix.
5. E NVIRONMENT
The basic environment is a 10 × 10 grid, with the goal state at the top left corner.
The agent spawns in a random non-goal state, and incurs a reward of − 1 for each time-step until it reaches the goal.
At the goal state, the agent no longer receives a loss and terminates the episode.
The agent is allowed ﬁve actions, A = { Up, Down, Left, Right, Stay } .
Moving into the wall is equivalent to executing a Stay action.
We choose this reward function for the beneﬁt of having a clear representation of the notion of “good enough”, which is reﬂected in how long it takes to reach the goal state.
Having such representation exempliﬁes how the APE can prevent an observer from cloning a good policy.
5. V ISUALIZATION
Figure 2 shows an example of a PE that is trained for the basic gridworld environment. Figureshows the corresponding cloned policy, as well as a comparison to an optimal policy.
The colour scale represents the expected return of starting at a given state.
In the case of an optimal policy ( β = 0 ), actions are taken to take the agent to the goal state as fast as possible.
However, when β > 0 , such a solution is no longer the optimum.
Similar to β = 0 , the experts would like to maximize the expected reward, and reach the goal state.
However, to minimize the reward of the observed policy, the two expert policies must jointly learn to increase the number of steps needed for π o to reach the goal state.
The expert policies must use adversarial behaviour while reaching the goal state, such as taking intelligent detours or Stay in the same state, which are learned to hinder π o as much as possible.
These learnt behaviours cause the cloned policy to take a drastically longer time to reach the goal.
For example, note the two purple squares at the top-left near the goal, which indicates that the experts understand that they should not move to prevent the observer from attaining reward.
Even though these sub-optimal decisions are made, on expectation, the experts are “not bad” and achieve an average of − 15 .
27 reward.
5. B ASELINES
We use behaviour cloning to clone our PGAPE trained policies.
To support our claims of preventing even in the horizon of inﬁnite data, we collect a million timesteps of the trained
PE in the environment.
Further details of behaviour cloning are in the appendix.
Shown in
Figure 3 is an optimal policy, and the resulting cloned policy from Section 5.1.
As well, we evaluate against other PE, to show that preventing against behaviour cloning is non-trivial.
We use several baselines.
We ﬁrst test policies that have approximately the same return as our ensemble by training PG with, and halting early rather than running until convergence.
In the Near-Optimal case, we ran until the PE had expected returns that matched the average achieved by our method.
Conversely, “Random” policies are used as a comparison to show that it is possible to cause the cloned policy to do poorly, but the tradeoff is that the PE itself cannot perform well, which is undesirable.
These policies are also policies trained with PG, except they are stopped much earlier, when their clones matches the expected returns of our PG-APE.
For each PG-APE, we use n = 2 different tabular policies treated as an ensemble, which we then clone, and average across 5 seeds. For the baselines, we hand-pick the policies, and thus only use 3 different policies.
[Table]
As presented in Table 1, all other PE have an insigniﬁcant difference (returns of the PE subtracted from returns of the cloned policy) between the performance of the PE and the cloned policy, except for our method.
These empirical ﬁndings show that preventing behaviour cloning difﬁcult, but possible using APE.
D ISCUSSION & F UTURE W ORK
Conﬁdential Policies: There are promising research directions regarding the protection of policies, due to the many applications where conﬁdentiality is crucial.
As long as there is a model of the observer, our presented method provides a worst-case scenario of experts.
In our work, we focused on the case where the observer does not use the current trajectory to determine their policy.
Instead, it may be the case that the observer uses a sequential policy (one that depends on its previous states and/or actions), such as an RNN to determine the context of the current expert.
Formally, the observer will no longer learn the policy formulated in Equation 1 that is solely dependent on the current state, but rather a policy that is dependent on the current trajectory:
[Equation]
We found in our preliminary results that using an RNN classiﬁer which outputs p ( c | τ 1: [Equation] ) simply ended up in with either optimal policies or crippled policies.
In both cases, there was a relatively minor difference in performance between the policy ensemble and the cloned policy.
Unsurprisingly, when the observer has access to a strong enough representation for their policy, then they should be able to imitate any policy .
In this case, the worst-case set of experts cannot do much to prevent the cloning.
We believe that this is an exciting conclusion, and is grounds for future work.
Continuous: Although our methods are evaluated in discrete state spaces, our approach can be generalized to continuous domains.
The Monte Carlo sampling in Equation 9 suggests that the use of continuous context may also be possible, given there is a strong enough function approximator to estimate the distribution of c | s .
We see this as an exciting direction for future work, to recover the full spectrum of possible adversarial policies under the constraint of Equation 2.
The Semantics of Reward: Although the minimization in Equation 2 implies a logical equivalence between the success of behaviour cloning to the reward the cloned policy can achieve, it may follow that this is not the case.
It may be the case that useless is deﬁned differently by the expected reward the cloned policy achieves on a different reward function ˜ r .
For example, a robot that is unpredictable should not be deployed with humans.
Since the r functions in Equation 2 are disentangled, the reward function r that is minimized in Equation 2 can be engineered to ﬁt any deﬁnition of uselessness.
We can modify the objective of APE by modifying Equations 4 and 5 to use a different reward function ˜ r in the minimization, substituting R ( t ) for ˜ R ( t ) = (cid:80) T − 1 t (cid:48)
=
t γ t (cid:48) − t ˜ r ( s t (cid:48) , a t (cid:48) ) .
The rest of the derivation and algorithm remain the same.
We think this is an exciting direction, especially for learning all different possible representations of the worst-case experts.
C ONCLUSION
We present APE as well as its mathematical formulation, and show that policy gradient, a basic
RL algorithm can be used to optimize a policy ensemble that cannot be cloned.
We evaluated APE against baselines to show that adversarial behaviour is not feasible without our method.
This work identiﬁes a novel yet crucial area in Reinforcement Learning, regarding the conﬁdentiality of proprietary policies.
The essence of our approach is that a policy ensemble can achieve high return for the policy owner, while providing an external observer with a guaranteed low reward, making proprietary ensemble useless to the observer.
The formulation of our problem setup and the algorithm are very general.
In this ﬁrst work we demonstrate the solution in the deliberately chosen simple environments in order to better visualize the essence of our method.
In our concurrent work we study thoroughly the application of our method in various domains, which is out of the scope of this introductory paper.
A A PPENDIX A. T RAINING D ETAILS & H YPERPARAMETERS
For our training, we set α θ = 0 . and the value weight to be 0 . 5 , use annealed entropy regularization (Mnih et al., 2016) from 5 e − 1 to 5 e − 3 and set the discount factor γ = 0 . 99 .
Due to the contrasting gradients experienced, large batch sizes are used.
In our experiments, we take 1 gradient update of
AdaM (Kingma & Ba, 2015) per batch of 4096 (containing multiple trajectories), and trained for 3 etimesteps.
To estimate p ( c | s ) in Equation 1, we use a replay buffer that keeps track of the previous 60 contexts seen at each state.
Estimating the quantity in Equation 8 requires memory, which we use a single GRU (Cho et al., 2014) as done in Strouse et al.
(2018), with the exception that only states are fed in as a one-hot.
Due to our environment is deterministic, state sequences captures the action sequence information.
The single unit is then concatenated with the state, which feeds into a fully connected layer of 128, and then a soft-max, to produce the distribution c | s over contexts.
For our behaviour cloning, we collect 1 e 6 state action pairs, and train a tabular policy with 0 .learning rate on cross entropy softmax loss for 100 epochs.
The large amount of data and epochs is to ensure that we can recover π o with little to no variance.
To solve the precise returns of the policies, we inject noise of 1 e − 9 , to ensure a hitting time always exists from each state.
As well, we clip all the hitting times to − T = − 100 .
A. E STIMATING ∇ θ log π o
It is not obvious how ∇ θ log
π o should be estimated, since π o is never realized until the policy is cloned.
Literally, it is a virtual policy.
Equation 1 offers a straightforward method to back-propagate, similar to that of the Mixture of
Experts model Jacobs et al.
(1991), except using an estimate of c | s instead of a gating network.
However, we can also rewrite Equation 1 as (cid:80) i p ( c ( i ) | s ) π c
( i ) ( a | s ) =
E c ∼ p ( c | s )
[ π c ( i ) ( a | s )] , which results in the gradient update being:
[Equation]
which suggests a method of Monte Carlo sampling the inner expectation with 1 sampled context.
Empirically, we use the Monte Carlo sampling method.