Paper: Analyzing the Effects of Emulating on the Reinforcement Learning Manifold (ICLR-2024)

Reviewer comment (Theory):
The connections between Lemmas, Propositions, Corollaries, and their proofs to the paper's central message are unclear.

Author response:
*“Throughout the paper, I find it challenging to link the various Lemmas, Propositions, Corollaries, and their proofs to the central message of the paper. For instance, the purpose and significance of Lemma 3.1 (and its proof) remain unclear to me. The necessity and meaning behind Propositions 3.3 and 3.5 are also ambiguous.”*  

Lemma 3.1, Proposition 3.3, and Proposition 3.5 together theoretically demonstrate  that learning only from expert trajectories results in highly non-robust policies in a natural setting. In particular, Lemma 3.1 proves that states in expert trajectories have large projections onto a subspace implicitly defined by the actions leading to high reward. Proposition 3.3 proves that training on states from the subspace described in Lemma 3.1. results in a policy that is essentially untrained (i.e. takes actions no better than random guessing) outside this subspace.  Proposition 3.5 then uses Proposition 3.3 to analyze a natural setup in which a policy trained in the subspace is highly non-robust. The non-robustness and vulnerabilities of these policies are further justified empirically in Section 4 and Section 5.

Reviewer comment (Reproducibility):
The experiments are missing crucial details, including specifics on the policy architecture, and the explanations of the results are unclear.

Author response:
**1.** *”While I can find the hyperparameters of the algorithms, there is no mention of the policy architecture. For instance, is a neural network used to represent the Q-network, and if so, how many layers does it have, and what are the dimensions of these layers?”*  

As also mentioned in the first paragraph of our supplementary material under the Section titled “HYPERPARAMETER DETAILS AND ARCHITECTURES”, the exact same hyperparameters as the original paper [1] are used and reported here. Furthermore, in Line 23 to 24 of our supplementary material it states that vanilla policies are trained with Deep Double Q-Network (DDQN). Hence looking at these lines it is clear that **indeed** neural networks are used to represent the Q-network. It can be seen from both of these citations provided that both of these algorithms use 3 layers of convolutional neural networks of size 32,64,64 respectively.  

[1] IQ-learn: Inverse soft-q learning for imitation. NeurIPS 2021.

Reviewer comment (Writing):
The paper intends to analyze the reinforcement learning manifold; however, the meaning is not evident from the text. Clarification is needed on what this entails and how the theoretical and empirical analyses relate to it.

Author response:
*“The paper intends to analyze the reinforcement learning manifold. However, the meaning is not evident from the text. Could you clarify what this entails and how the theoretical and empirical analyses relate to it?”*  

The reinforcement learning manifold is represented by the graph of the function $J_{\pi}(s)$, and thus implicitly given by the policy $\pi$.  
Here $J_{\pi}(s)$ is the expected cumulative rewards obtained by policy $\pi$ starting from state $s$.  
The paper both theoretically and empirically analyzes how small perturbations from states $s$ on the learned trajectory yield large changes in $J_{\pi}(s)$ when $\pi$ is trained from expert trajectories, equivalently corresponding to the manifold represented by $J_{\pi}(s)$ having steeper slopes near the learned trajectory.

Reviewer comment (Writing):
In several sections, the paper mentions "with reward" and "without reward," but their exact meanings are unclear. It is uncertain whether "without reward" refers to imitation and inverse RL, and clarification is needed, especially since "without reward" is commonly associated with unsupervised RL. Consistency in maintaining this distinction throughout the paper is crucial.

Author response:
*“In several sections, the paper mentions "with reward" and "without reward." However, their exact meanings are unclear. Does "without reward" refer to imitation and inverse RL?”*  

Yes, without reward refers to imitation and inverse RL.  

*”Furthermore, "without reward" is commonly associated with unsupervised RL, where the agent does not receive a reward signal. Could you please provide clarification? “*  

Yes, as mentioned above without reward refers to imitation and inverse reinforcement learning.

Reviewer comment (Writing):
Throughout the paper, several instances of vague language make comprehension challenging. For instance, clarification is needed on whether “that focus on learning via emulating affect” refers to imitation learning, and what exactly is meant by "learning from exploration"—is it referring to regular reinforcement learning?

Author response:
*“For instance, does “that focus on learning via emulating affect” refer to imitation learning?”*  

Yes, it refers to imitation learning.  

*“What exactly is meant by "learning from exploration"? Is it referring to regular reinforcement learning?”*  

Yes, learning from exploration refers to regular reinforcement learning.

Reviewer comment (Writing):
The statement "Our paper is the first to focus on the adversarial vulnerabilities of deep neural policies that can learn without a reward function" requires clarification on how the proposed method learns without a reward function.

Author response:
*“Could you clarify how the proposed method learns without a reward function?”*  

The method mentioned here is the inverse Q-learning algorithm which is also explained in Page 3 between the lines 12 and 19 of the Section titled “Learning without Rewards”.

Reviewer comment (Theory):
It is challenging to link the various Lemmas, Propositions, Corollaries, and their proofs to the central message of the paper. For example, the purpose and significance of Lemma 3.1 (and its proof) remain unclear, and the necessity and meaning behind Propositions 3.3 and 3.5 are ambiguous. Clarification is also needed on the intent of Definition 4.1 and how it relates to the main objective of the paper.

Author response:
*“Throughout the paper, I find it challenging to link the various Lemmas, Propositions, Corollaries, and their proofs to the central message of the paper. For instance, the purpose and significance of Lemma 3.1 (and its proof) remain unclear to me. The necessity and meaning behind Propositions 3.3 and 3.5 are also ambiguous.”*  

Lemma 3.1, Proposition 3.3, and Proposition 3.5 together theoretically demonstrate  that learning only from expert trajectories results in highly non-robust policies in a natural setting. In particular, Lemma 3.1 proves that states in expert trajectories have large projections onto a subspace implicitly defined by the actions leading to high reward. Proposition 3.3 proves that training on states from the subspace described in Lemma 3.1. results in a policy that is essentially untrained (i.e. takes actions no better than random guessing) outside this subspace.  Proposition 3.5 then uses Proposition 3.3 to analyze a natural setup in which a policy trained in the subspace is highly non-robust. The non-robustness and vulnerabilities of these policies are further justified empirically in Section 4 and Section 5.

Reviewer comment (Reproducibility):
Implementation details are missing from the paper. Specifically, there is no mention of which architecture is used to represent the policy or which specific algorithms are utilized for IRL and imitation. Without these specifics, it is challenging to justify the performance presented.

Author response:
**1.** *”While I can find the hyperparameters of the algorithms, there is no mention of the policy architecture. For instance, is a neural network used to represent the Q-network, and if so, how many layers does it have, and what are the dimensions of these layers?”*  

As also mentioned in the first paragraph of our supplementary material under the Section titled “HYPERPARAMETER DETAILS AND ARCHITECTURES”, the exact same hyperparameters as the original paper [1] are used and reported here. Furthermore, in Line 23 to 24 of our supplementary material it states that vanilla policies are trained with Deep Double Q-Network (DDQN). Hence looking at these lines it is clear that **indeed** neural networks are used to represent the Q-network. It can be seen from both of these citations provided that both of these algorithms use 3 layers of convolutional neural networks of size 32,64,64 respectively.  

[1] IQ-learn: Inverse soft-q learning for imitation. NeurIPS 2021.

Reviewer comment (Theory):
The case of studied linear Q-function approximators in the theoretical part is quite restrictive. The assumptions used for Proposition 3.5 are very strong. A potentially more general direction would be to study smooth Q-approximators in the case of Lipschitz MDPs.

Author response:
*“The case of studied linear Q-function approximators in the theoretical part is quite restrictive. The assumptions used for Proposition 3.5 are very strong. A potentially more general direction would be to study smooth Q-approximators in the case of Lipschitz MDPs.”*

An important point here is that the theoretical part is demonstrating a failure mode of training from expert trajectories. Typically this is done by producing a counter-example MDP in which the failure occurs. Thus, a theoretical demonstration of such a counter-example in a simple setting is a stronger result, as it demonstrates that the algorithm fails even in very simple MDPs. MDPs with bounded, linearly parameterized rewards and transitions are immediately Lipschitz, and thus our counter-example setting does indeed correspond to a Lipschitz MDP. 

Furthermore, linear function approximation is currently at the frontier of research that develops a rigorous theoretical understanding of reinforcement learning algorithms [1,2,3,4,5,6,7,8]. Thus, our theoretical results are demonstrated in the most general setting currently amenable to provable mathematical analysis. 

[1] Bilinear classes: A structural framework for provable generalization in RL. ICML, 2021.

[2] Reinforcement learning from partial observation: Linear function approximation with provable sample efficiency. ICML 2022.

[3] Provably efficient reinforcement learning with linear function approximation. Conference on Learning Theory. PMLR, 2020.

[4] Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. Conference on Learning Theory. PMLR, 2021.

[5] Learning near optimal policies with low inherent bellman error. ICML, 2020.

[6] Reinforcement learning in linear MDPs: Constant regret and representation selection. NeurIPS 2021.

[7] Nearly minimax optimal reinforcement learning with linear function approximation. ICML, 2022.

[8] First-order regret in reinforcement learning with linear function approximation: A robust estimation approach. ICML 2022.

Reviewer comment (Experiments):
Another weakness is that the authors do not offer no solutions to the problem of lack of robustness. This could be in the form of better training methods for the policy like regularization or better strategies to collect demonstrations.

Author response:
*“Another weakness is that the authors do not offer no solutions to the problem of lack of robustness. This could be in the form of better training methods for the policy like regularization or better strategies to collect demonstrations.”*

Please note that in the adversarial machine learning literature there is a line of work dedicated solely to discussing the explicit and concrete vulnerabilities of the models [1,2,3,4,5,6,7], and there is a line of work dedicated to addressing these problems described in these studies [8,9]. We believe these two lines of research progress side by side, without the expectation on one single paper carrying both of these perspectives at the same time.


[1] On Adaptive Attacks to Adversarial Example Defenses, NeurIPS 2020.

[2] Label-Only Membership Inference Attacks, ICML 2021.

[3] Poisoning and Backdooring Contrastive Learning, ICLR 2022.

[4] Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness, AAAI 2023.

[5] EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples, AAAI 2018.

[6] Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs, AAAI 2022.

[7] Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning, AAAI 2020.

[8] Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations, NeurIPS 2020.

[9] Robust Deep Reinforcement Learning through Adversarial Loss, NeurIPS 2021.