Mitigation of Adversarial Policy Imitation via Constrained Randomization of Policy (CRoP)
Abstract
Deep Reinforcement Learning (DRL) policies are vulnera- ble to unauthorized replication attacks, where an adversary exploits imitation learning to reproduce target policies from observed behavior.
In this paper, we propose Constrained Randomization of Policy (CRoP) as a mitigation technique against such attacks.
CRoP induces the execution of sub-optimal actions at random under performance loss constraints.
We present a parametric analysis of CRoP, address the optimality of CRoP, and establish theoretical bounds on the adversarial budget and the expectation of loss.
Furthermore, we report the experimental evaluation of CRoP in Atari environments un- der adversarial imitation, which demonstrate the efficacy and feasibility of our proposed method against policy replication attacks.
Introduction
Deep Reinforcement Learning (DRL) is a learning framework for sequential decision-making leveraging neural networks for generalization and function approximation.
With the growing interest in DRL and its integration in commercial and critical systems, the security of such algorithms have become of paramount importance (Behzadan and Munir 2018).
In tandem with DRL, similar advancements have been made in Imitation Learning (IL) techniques that utilize expert demonstrations to learn and replicate the expert’s behavior in sequential decision making tasks.
Deep Q-Learning from
Demonstration (DQfD)(Hester et al. 2017) is an IL variant that has enabled DRL agents to converge quicker to an optimal policy.
However, recent work in (Behzadan and Hsu 2019a) and (Chen et al.
2020) demonstrate that IL can also be exploited by adversaries to replicate other agents’ policies from passive observation of their behavior.
This gives rise to risks concerning intellectual property and adversarial information gain for more effective active attacks.
Current state of the art in countering such attacks include watermarking (Behzadan and Hsu 2019b)(Chen et al. 2021), which enables the post-attack identification of replicated policies.
In this paper, we propose an active mitigation technique against policy imitation attacks, named Constrained Randomization of Policy (CRoP).
The proposed technique is based on intermittent randomization of a trained policy, constrained on a threshold for maximum amount of acceptable loss in the expected return.
The goal is to increase the adversary’s imitation training cost, measured as the minimum number of training iterations and observed demonstrations required for training a replica that matches the target policy’s performance.
The main contributions of this paper are: (1) We propose and formulate CRoP as a mitigation technique against adversarial policy imitation, (2) We present a formal analysis of the bounds on expected loss of optimality under CRoP, (3)
We formally establish bounds on the adversary’s imitation cost induced by CRoP. (3)
We report the results of empirical evaulation of adversarial imitation via DQfD against CRoP agents in classical DRL benchmarks, and demonstrate the efficacy and feasibility of CRoP in those settings.
The remainder of this paper is organized as follows: we introduce CRoP as a mitigation technique against policy imitation, and analyze the optimality of a CRoP policy in relation to an optimal policy. The analysis further establishes a lower bound on adversarial budget induced by CRoP.
The following section reports the experimental evaluation of CRoP in three
Atari benchmark environments, along with measurements of the training and test-time performance of DQfD-based adversarial imitation learning agents targeting CRoP-enabled policies.
We conclude the paper with a summary of findings and remarks on future directions of research.
Constrained Randomization of Policy
In the remainder of this paper, we assume the target policy aims to solve a Markov Decision Process (MDP) denoted by the tuple < S, A, R, T, γ > where S is a finite state space,
A is a finite action space, T defines the environment’s transition probabilities, a discount value γ ∈ [0 , 1) , and a reward function R : S ×
A → [0 , 1] .
The solution to this
MDP is a policy π : S
→ A that maps states to actions.
An agent implementing a policy π can measure the value of a state V ( s ) = max a ( r s,a + γV ( s ′ ) ), where s ′ is the next state.
Similarly, the value of a state-action pair is given by
Q ( s, a ) = max a ( r s,a + γQ ( s ′ , a ′ )) where s ′ is the next state and a ′ is the next action.
Constrained Randomization of Policy (CRoP) is an action diversion strategy from an optimal policy under constrained performance deviation from optimal.
Let ˆ a ∈ ˆ
A where ˆ a are candidate actions that satisfy | Q ( s, π ( s ))
− Q ( s, ˆ a i ) | < ρ .
In other words, ˆ A is the space of all candidate actions for s ∈ S excluding the optimal action π ( s ) .
We define CRoP as the function below:
[Equation]
Where U ( ˆ A ) is the uniform distribution over ˆ A .
This definition of ρ threshold is the difference of Q-values.
We have three variations of ρ for CRoP: Q-value difference (Q-diff) as described in Equation 1, and two measures inspired by the advantage function: advantage-inspired difference (A-diff), and positive advantage-inspired difference (A + -diff).
A-diff
CRoP is thus defined as:
[Equation]
A + -diff’s ρ has the condition ˆ A [Equation] s t , a t ) ≥ 0 .
A-diff and A + - diff’s ρ are interpreted as 1-step hindsight estimation which is relevant to the trajectory taken instead of only pure future estimate as with Q-diff, eg.
played badly, now play safe vs. plan to feint ahead.
However, the selection of ρ should consider estimation error due to either finite training or function approximation.
One can look to the analysis of learning complexity as a method of finding error bounds to derive a safety margin for ρ .
We choose these three threshold variations because their performance vary across the environments, implying that being able to successfully deviate from optimal policy is conditioned on the environment dynamics itself as well as the defender’s tolerance for loss which may not be captured by a single threshold such as Q-diff’s.
We cannot use the traditional Advantage since V ( s ) = max a Q ( s, a ) implies Q-diff’s implementation would have a similar impact in regards to threshold.
Additionally, it is important to recognize that CRoP is similar to that of a ϵ -greedy policy; however, the difference lies on the constraint expected loss that ϵ -greedy does not guarantee.
[Figure]
By definition, a policy π is ϵ -optimal if there exists a nonnegative constant ϵ such that v π ( x ) ≥ V ( x ) − ϵ for all initial states x in S .
Some definitions include that it occurs at P (1 − δ ) , which we will abide by.
In other words, ϵ -optimal policies are within an ϵ neighborhood of V ∗ , specifically V ∗ − V π < ϵ for all a ∈ A and s ∈ S at probability (1 − δ ) .
As illustrated in Figure 1, π ∗ is the optimal and greedy policy extracted from V ∗ where π is the extracted policy from V π and π ′ is the extracted policy from V π ′ , we see that π ′ may be expressed as an ( ϵ + ϵ ′ )
-optimal to Q ∗ / V ∗ when evaluated for the initial states and to follow a greedy policy thereafter.
Since we do not assume π to be an optimal policy, it is possible for π ′ to be more optimal than π .
However, it is noteworthy that an evaluation of optimality based on the difference to the value function does not imply extracted policies with small error to V ∗ resemble the optimal policy when assessed on behavioral differences.
Theorem1 establishes that CRoP policy f is at worst ( ϵ + ϵ ′ )
-optimal to Q ∗ at probability (1 − δ ) as an evaluation of the initial states assuming a greedy policy is followed after.
However, an evaluation for committing to following CRoP thereafter (or for any evaluation of a trajectory under CRoP via sequence of Q-values) will have compounding sub-optimality or horizon dependent error. However, ϵ -optimal can have compounding error in expectation regardless of CRoP.
Therefore, instead for any fix length horizon T , we know that π ′ will be ( T × ϵ + ϵ ′ )
-optimal to the sum of the taken trajectory’s states taken under π ′ evaluated by V ∗ . In other words,
[Equation] for a finite state trajectory set s ∈ S of cardinality T under π ′ ; however, this is different than a trajectory taken under π ∗ .
We cannot evaluate an optimal trajectory from local view; however, since we allow the defender to modify ρ , one can simply stop the deviating behavior to cease and bound the compounding error at state s , though the error in a trajectory measured by difference in the sum of Q-values from Q ∗ may continue to compound.
Theorem 1 Given Q ∗ ( s t , a t ) − Q π ( s t , a t ) <
ϵ ′ at probability (1 − δ ) and | Q π ( s t , a t ) − Q π ′ ( s t , a t ) | ≤ ϵ for all s ∈ S and a ∈ A , then Q ∗ ( s t , a t ) − Q π ′ ( s t , a t ) ≤ ϵ + ϵ ′ at probability (1 − δ ) .
π ′ is an ( ϵ + ϵ ′ )
-optimal to Q ∗ /V ∗ at probability (1 − δ ) .
Proof.
Given Q ∗ ( s t , a t ) − Q π ( s t , a t ) <
ϵ ′ at probability (1 − δ ) and | Q π ( s t , a t ) − Q π ′ ( s t , a t ) | ≤ ϵ for all s ∈ S and a ∈ A , then Q ∗ ( s t , a t ) − Q π ′ ( s t , a t ) ≤ ϵ + ϵ ′ at probability (1 − δ ) .
Let Q diff =
Q ∗ ( s t , a t ) − Q f
( s t , a t ) + | Q f ( s t , a t ) −
Q π ′ ( s t , a t ) | .
Given that Q ( s, a ) ∈ (0 , 11 − γ ) , at (1 − δ ) probability:
Q ∗ ( s t , a t ) − Q π ′ ( s t , a t ) ≤ Q diff ≤ ϵ + ϵ ′ (
[Equation]
We consider two common approaches to IL: (1) Behavioral Clones (BC) which are supervised learners, and (2) reinforcement learning from demonstration techniques such as DQfD, which augment RL with IL.
Work by (Ke et al. 2020) shows that: BC minimizes the KL divergence, Generative Adversarial Imitation Learning (GAIL)
(Ho and Ermon 2016) minimize the Jensen Shannon divergence and DAgger (Ross, Gordon, and Bagnell 2011) minimizes total variance.
For BC, CRoP affects the maximum likelihood in a similar manner to data poisoning attacks like label flipping (Xiao, Xiao, and Eckert 2012) or class imbalance.
In regard to GAIL, the discriminator from a GAN prioritizes expert experiences so unless modified for decay when out-performed, additional penalty is given to the training policy.
Furthermore, when CRoP lowers the action distribution for a ∗ according to δ probability and increases the distribution for candidate actions, it results in smaller maximal difference for DAgger. [Figure] [Figure]
[Figure]
E p i s ode R e w a r d
[Figure]
Budget Analysis for Perfect Information Adversary
The adversary objective is one of two objectives: (1) policy replication such as behavioral imitation or structural similarity in learned parameters, (2) training computation cost reduction.
A defender’s objective is to minimize adversarial gain or maximize the adversary’s budget while simultaneously maximizing their protected policy’s return; however, since the minimization of adversarial gain results in performance deterioration, a defender must also determine the tolerance of performance deterioration.
We measure the adversary’s budget in the sample quantity or trajectories that it can acquire through a passive attack.
Nair and Doshi-Velez (Nair and Doshi-Velez 2020) derive upper and lower bounds on the sample complexity of direct policy learning and model-based imitation learning in relaxed problem spaces.
This follows the research of RL sample efficiency and Offline RL(Levine
et al. 2020).
However, in this work we divert from a direct treatment of sample efficiency to consider information optimality from observed target demonstration without environment interaction.
Consider the set T where τ i ( ∀ , τ i ∈ T ) which is composed of a T -length chain of ( s, a ) -pairs.
Assume each ( s, a ) -pair has two possible outcomes, optimal at P ( δ ) or sub-optimal at P (1 − δ ) .
Assume pair and trajectory uniqueness, this would contain 2 T trajectories where T is the length of the horizon.
To obtain optimal target π , we would require all trajectories except the event of a complete sub-optimal trajectory (1 − δ ) T .
Let τ w be the worst-case trajectory and ˆ m be the sum of the expected number of trajectories for each sequential pull from T .
It follows that:
[Equation]
Intuitively, we see in the denominator the probability of pulling unseen trajectories given the trajectories in ˆ T and known probability for all τ
i ∈
ˆ T .
We know an expectation on expensive to obtain informative trajectories from π .
However, typically an adversary has a fixed budget and therefore we would want to know what to expect given their budget B , here we calculate for a budget measured in optimal stateaction pairs.
To calculated an expected number of optimal state-action pairs, we find a t < T such that:
[Equation]
Given we can reset to the previous state and resample until we obtain an optimal state-action pair.
This would give an expectation for the adversary to obtain t optimal state-action pairs with B budget.
This can be extended to the expectation of number of trajectories by approximating B , similar to
Equation 5 where we find a t < T , but with Equation 4.
Policy Evaulation and Expectation of Loss
We see that the Q-value under f will be either equivalent or less than the Q-value under target policy π which dictates selected a ′ .
Furthermore, the expected return G ft for stochastic policy f with uniform sampling from ˆ A is expressed as the following:
[Equation]
With Equation 6, G ft is the weighted sum of an optimal expected return at probability δ and the expected return across all rewards given by candidate actions at probability (1 − δ ) .
Given G ∗ t and G ft , the difference between the expected return in Q -value form is exactly:
[Equation]
[Equation] Since Q π ( s t , a t ) − E [ Q f ( s t , ˆ a t )] < ρ , then the expectation loss G ∗ t − G ft ≤ (1 − δ ) ρ ≤ ρ .
This expectation of loss is calculated from the current state’s forward estimation of future reward.
We see there exists an upperbound, call it E [ L ] which is derived from the evaluation of trajectory returns taken under f which is the agent’s local reference.
This is important to defenders because they can estimate when to cease CRoP given its local view:
[Equation]
Experimental Evaluation
We investigate DQfD as our adversarial IL method and evaluate test-time and training time performance across three
Atari environments: Breakout, Cartpole, and Space Invaders.
We train DQfD agents under default parameters (supplied in supplements) with CRoP induced demonstrations, a control DQfD agent (our baseline IL comparison), and a default, double DQN (DDQN) agent which provided the expert demonstrations and as well as be a baseline performance comparison for IL agents.
The results of a parameter search on trained DDQN policies from Stable-Baseline Zoo (Raffin 2018) and the count of candidate timesteps are presented in the longer extension of this paper.
The test evaluations of the target policy under selected thresholds of CRoP follow in Figure 2 to show that the selected ρ and δ values reflect the evaluation of average reward performance.
The trade-off on δ and ρ is similar to an allowance of high or low variance in
Q-value.
One can draw a similarity of the defender’s selection of δ and ρ as risk-adverse, risk-neutral, and risk-seeking behaviors determined by the defender.
The results, illustrated in Figure 4 , demonstrate that the performance of imitated policies generally remain below their control/baseline DQfD agents for earlier spans of training episodes.
We compare it to the baseline DQfD because the baseline demonstrates the performance of a DQfD agent with no mitigation against adversarial ease-dropping on state-actions performed by the target agent.
CRoP may induce variance similar to optimistic initialization, for example, work by (Kamiura and Sano 2017) and (Szita and Lörincz 2009).
However, we argue that by adding deviating behavior to the assumed optimal target policy, the target policy is withholding the maximal information gain an adversary can observe, thus increasing their adversarial budget.
Figure 3 depicts the comparison of test-time performance among agents trained with various values of δ and ρ .
We emphasize the constrains in CRoP are expected loss which are not true performance loss.
This supports the need for several threshold variations because policies under CRoP behaved differently across the environments.
Conclusion
This study investigated the threat emanating from passive policy replication attacks through adversarial usage of Imitation
Learning.
We proposed Constrained Randomization of Policy (CRoP), a deviation from optimal policy under a threshold constraint, as a mitigation technique against such attacks.
We perform a parameter search and empirically evaluate the target policy under CRoP in comparison to the target policy without protection.
We analyzed its performance with regards to ϵ -optimality, estimated impact on adversarial cost and the expectation of loss.
Furthermore, we empirically evaluated
CRoP across 3 Atari game benchmarks and verified the efficacy and efficiency of CRoP against DQfD-based policy replication attacks, demonstrating that it is possible for the target policy to accomplish its task while deviating behavior in a bounded manner to increase the adversarial cost for successful policy replication.