Paper: Stealthy Imitation: Reward-guided Environment-free Policy Stealing (ICLR-2024)

Reviewer comment (Experiments):
Policy stealing is performed on simplistic mujoco case studies with simple MLPs. While the threat model is realistic, the application seems simple. Moreover, there are no experiments with realistic robotic imitation learning datasets (e.g. robomimic, D4RL's Adroit, CARLA) and recent models (e.g. diffusion policies, behavior transformers). This is particularly important given that model stealing is particularly relevant with large models only available as API such as the recent RT-2-X model [1]. The authors provide an abltaion with number of layers in an MLP but not with other architectures.

Author response:
> 1. While the threat model is realistic, the application seems simple.

We add new experiments to validate Stealthy Imitation in realistic scenarios where the victim policies are trained for a Franka Emika Panda robot arm provided by panda-gym [1]. Please see our [general comment about Validation of Applicability in Realistic Scenarios](https://openreview.net/forum?id=7q7s5fXEpP&noteId=ebtZUbFh9r) for a summary. We have also updated the paper draft to include the experiment details on the Panda tasks, which you can find in Appendix G.


> 2. This is particularly important given that model stealing is particularly relevant with large models only available as API such as the recent RT-2-X model. 

We want to clarify that image models such as RT-2-X do not fit the setup of the knowledge-limited attacker we consider.
Particularly, the input range is known for the image domain (i.e., pixel values in [0,255]). For this case, classic existing model stealing methods can be applied.

Reviewer comment (Experiments):
What would the required budget be to scale to image or lidar inputs?

Author response:
> 5. What would the required budget be to scale to image or lidar inputs?

As clarified previously, the assumptions of an unknown input range to the attacker does not hold for images or lidar. As such, this extension is not within the scope of the paper.

Reviewer comment (Evaluation):
Is there perhaps a way to break the defense along the lines of adaptive attacks proposed in [2]?

Author response:
> 6. Is there perhaps a way to break the defense along the lines of adaptive attacks proposed in (Tramer et al. "On Adaptive Attacks to Adversarial Example Defenses." 2020.)?

We would like to emphasize that [Tramer et al., 2020] and other results in that line of work address adversarial examples, a different type of security threat than model stealing; as such, these are not directly applicable to our setup.

Nevertheless, we can consider the consequences of the attacker becoming aware of the defense strategy.
In that case, our defense can be enhanced to produce responses with similar variance for queries outside the range as those within range.
For more details, please also see our [answer to Reviewer 3FCu's 4th point](https://openreview.net/forum?id=7q7s5fXEpP&noteId=XiSDuxa5Ma) on possible attack variations.

Could the reviewer let us know if these variations answer the scenarios they had in mind?

Reviewer comment (Writing):
The writing could be clearer in some places. The proposed algorithm has many components, and some of the experiments are somewhat complex—it was a bit hard to understand the purpose of some algorithm components or experiments at first.

Author response:
We appreciate the suggestions for improving the clarity of the paper and have incorporated them into our revised draft.

Reviewer comment (Experiments):
The setting of wanting to steal a policy without knowing the environment seems unrealistic—what is the attacker planning to do with the policy if it doesn't have access to the environment? Wouldn't the point of stealing a policy be to run the policy in the environment, which entails the attacker has access?

Author response:
We consider two reasons why an attacker might seek to steal a policy without knowledge of the environment:

Firstly, **further attacks**: stealing a policy enables the attacker to disrupt the target control system. They can generate adversarial signals with the stolen policy, even without understanding the environment.

Secondly, **sensitive information exposure and unauthorized usage**: it enables the attacker to access confidential information, like sensor types or preprocessing methods. The information could potentially aid in the development of their own control system.

Reviewer comment (Experiments):
The comparison to DFME might not be fair—see my questions below.

Author response:
We aligned DFME's initial state output distribution with ours $ \mathcal{N}(\mathbf{0}_n,\mathbf{1}_n^2)$. The original DFME generator used a tanh activation function, confining outputs to [-1,1], typical in image classification model stealing. We adapt DFME to the best of our ability to the present setup by substituting this with **batch normalization with learnable scaling and shift factors**, enabling the generator to explore regions beyond the $ \mathcal{N}(\mathbf{0}_n,\mathbf{1}_n^2)$ starting point.

Additionally, we follow the reviewer's suggestion and amplify the output of the DFME generator by factors of 10 and 100. The resulting return ratio of the attacker policy in these two additional experiments is worse than without amplification throughout 50M queries.

Reviewer comment (Experiments):
The test environments are quite limited. These Mujoco environments are small, and there are only three of them. The only contact dynamics are with the ground and self-collision. Testing in larger environments with more degrees of freedom and richer dynamics is highly encouraged.

Author response:
We add new experiments to validate Stealthy Imitation in realistic scenarios where the victim policies are trained for a Franka Emika Panda robot arm provided by panda-gym [1]. Please see our [general comment about Validation of Applicability in Realistic Scenarios](https://openreview.net/forum?id=7q7s5fXEpP&noteId=ebtZUbFh9r) for a summary. We have also updated the paper draft to include the experiment details on the Panda tasks, which you can find in Appendix G

Reviewer comment (Theory):
This method seems impossible for policies with high-dimensional input such as images.

Author response:
We want to clarify that image data does not fit the setup of the knowledge-limited attacker we consider.  
Particularly, the input range is known for the image domain (i.e., pixel values in [0,255]). For this case, classic existing model stealing methods can be applied.

Reviewer comment (Experiments):
The approximation of the state distribution using a normal distribution seems quite limiting, and it's an open question as to whether this would work for more complex high-dimensional problems. The authors discuss this in 5.3 and provide some empirical experiments purporting to show that this is fine, but I am somewhat skeptical about the situation in problems with larger state spaces.

Author response:
We argue that the capacity to estimate states based on a normal distribution is a feature of our method.  
We show that this assumption is sufficient to approximate more complex distributions; this finding is part of the paper's contributions.  
The experimental results on the Panda robot arm show that this assumption holds for high dimensional state spaces (26 dimensions for the Panda Pick And Place task).

Reviewer comment (Evaluation):
The primary method proposed here seems like it would be prone to distribution shift over time. If I understand correctly, during each iteration, $\pi_a$ is trained on $D_v$ which consists of only the most recently collected data. At the same time $\mu$ and $\sigma$ are estimated by approximating the distribution of points where $\pi_a$ struggles to match the output of the victim $\pi_v$. It seems likely in this scenario that because the distribution that $\pi_a$ is trained on shifts over time (as $\mu$ and $\sigma$ change), it will get better at the regions of space most recently encountered, and may lose capability on regions that have not been visited recently. It seems that the use of $\pi_e$ which is reinitialized every time is meant to address this, but some discussion of the role of drift would be interesting here.

Author response:
We believe the reviewer's understanding is correct. We clarify that the distribution shift across iterations is indeed intentional and a key part of our method in search of the correct victim distribution. The evaluator $\pi_e$ plays a crucial role in selecting the most suitable distribution. The final attacker policy only needs to perform well on the distribution chosen by $\pi_e$, not on all explored distributions.

Reviewer comment (Evaluation):
It also seems that there are a lot of cat-and-mouse games that could be played with the proposed defense against this stealing technique as well. The authors suggest randomizing outputs for states that are outside of a known input range, but this is trivially circumvented by querying the same point multiple times and assuming points with low variance are within the training distribution (a lot of this depends on whether the policy is known to be stochastic). If the defender then tries to keep out-of-bounds points the same or close across multiple queries, then the attacker could start querying very close points. The defender could then try to build some random, yet fixed, yet smoothly changing landscape of points outside the boundaries, but this seems like a complicated modelling task. In the end, the suggestion of returning random points in the out-of-bounds range seems right, but like most security challenges, hard to get right in a way that prevents further circumvention.

Author response:
We agree with the reviewer that tackling security topics often involves a cat and mouse game between the attacker and the defender.  
Building on the reviewer's attack suggestion, our defense can be enhanced to produce responses with similar variance for queries outside the range as those within range. For instance, the defender could map queries outside the range to a random or fixed point within the range, resulting in a similar output variance. We appreciate this discussion and will incorporate it into the revised draft of our paper.

Reviewer comment (Experiments):
The authors target interactive policies in RL domains, but there doesn't seem to be anything here requiring RL, or specific to the RL setting. This technique should be applicable in any domain where you are trying to clone a network with input small enough that it can be modeled using a normal distribution. With that in mind, it would be good to see results on a wider variety of problems.

Author response:
We thank the reviewer for highlighting the general potential of our method. We include here results for Stealthy Imitation applied to binary classification on the Diabetes dataset [2]. The input consists of eight variables and has different semantics and scales; the output can be considered as a continuous "action" between [0,1]. We initialize the estimated distribution with $\mathcal{N}(\mu^*+3\sigma^*,(\sigma^*)^2)$, instead of $ \mathcal{N}(\mathbf{0},\mathbf{1})$ which results in high $D_{\mathrm{KL}}$ (2712). The accuracy of the victim model is 77\%. The final attacker model can achieve 69\% accuracy (90\% w.r.t. the victim) and the $D_{\mathrm{KL}}$ also has a decreasing trend as shown below.  

| Iteration         | 0   | 2   | 4   | 6   | 8   | 10  | 12  |
|-------------------|-----|-----|-----|-----|-----|-----|-----|
| $D_{\mathrm{KL}}$ | 4.5 | 4.4 | 4.0 | 3.6 | 3.2 | 3.1 | 3.1 |

While we acknowledge the potential of Stealthy Imitation beyond RL contexts (e.g., feature vectors, multivariate time series), we leave the in-depth analysis of these applications to future work.

Reviewer comment (Evaluation):
Addressing the questions about drift in the Weaknesses section would be great.

Author response:
We believe the reviewer's understanding is correct. We clarify that the distribution shift across iterations is indeed intentional and a key part of our method in search of the correct victim distribution. The evaluator $\pi_e$ plays a crucial role in selecting the most suitable distribution. The final attacker policy only needs to perform well on the distribution chosen by $\pi_e$, not on all explored distributions.

Reviewer comment (Evaluation):
Discussion of the issues with the counter-attack would also be beneficial.

Author response:
We agree with the reviewer that tackling security topics often involves a cat and mouse game between the attacker and the defender.  
Building on the reviewer's attack suggestion, our defense can be enhanced to produce responses with similar variance for queries outside the range as those within range. For instance, the defender could map queries outside the range to a random or fixed point within the range, resulting in a similar output variance. We appreciate this discussion and will incorporate it into the revised draft of our paper.

Reviewer comment (Experiments):
Transitioning from a controlled experimental setup to real-world applications might present unforeseen challenges, for example, the authors only experiment on simple tasks that have relatively simple state distribution. The underlying challenges for applying such tasks to a more complicated task is unclear.

Author response:
We add new experiments to validate Stealthy Imitation in realistic scenarios where the victim policies are trained for a Franka Emika Panda robot arm provided by panda-gym [1]. Please see our [general comment about Validation of Applicability in Realistic Scenarios](https://openreview.net/forum?id=7q7s5fXEpP&noteId=ebtZUbFh9r) for a summary. We have also updated the paper draft to include the experiment details on the Panda tasks, which you can find in Appendix G.

Reviewer comment (Novelty):
The reviewer has major concerns regarding the novelty of the methods, as model stealing with a synthetic dataset and a surrogate model (here state estimators) is not new to the community. The reviewer would appreciate if the authors could elaborate more on the challenges of this matter, especially the unique challenges when applying such attack in the RL setting.

Author response:
We would like to emphasize that our method differs from standard model stealing by focusing on how to estimate the valid input range, while we do follow the standard way to steal policy after the estimation process. Additionally, if by "state estimators" here the reviewer refers to the estimated probability distribution, then it is also different from previous works which train a generator model through back-propagation. We also include a reward model which helps understand the imitation difficulty across regions. For a detailed explanation on how our method contrasts with DFME, please also see the answer to the next point.

Reviewer comment (Novelty):
The baseline of DFML is not fair enough given the different settings here. Following the philosophy of DFML, it seems that the proposed SI is the same approach under the RL setting.

Author response:
Stealthy Imitation is the first method to steal a policy without the knowledge of input range, which means that there are no baselines for this setup. However, we want to provide a comparison to prior art and adapt DFME to the best of our ability to the present setup. 
We also summarize the difference between DFME and Stealthy Imitation in the following table; the two methods tackle different threat models and achieve different goals.  


| Method         | Data Source              | Target Feature            | Optimization Focus                            | Reusable                                                                     | Key Advantage                                                 |
|----------------|--------------------------|---------------------------|------------------------------------------------|---------------------------------------------------------------------------------|---------------------------------------------------------------|
| DFME    | Model-generated data     | Adversarial examples      | Sample difficulty with $L_1$ loss | Not, as adversarial examples vary from model to model              | Refines data distribution using knowledge of input range |
| Ours   | Probability distribution | Hard-to-Imitate Region   | Overall difficulty with reward model | Yes, as the input range does not change           | Efficiently estimates input range                 |

Reviewer comment (Experiments):
The reviewer has concerns regarding the complexity of the target RL tasks. While it might be easy to build a synthetic dataset for simple tasks like Hopper, it might be infeasible for more complicated tasks as the state dimension increases.

Author response:
We add new experiments to validate Stealthy Imitation in realistic scenarios where the victim policies are trained for a Franka Emika Panda robot arm provided by panda-gym [1]. Please see our [general comment about Validation of Applicability in Realistic Scenarios](https://openreview.net/forum?id=7q7s5fXEpP&noteId=ebtZUbFh9r) for a summary. We have also updated the paper draft to include the experiment details on the Panda tasks, which you can find in Appendix G.