Paper: Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR-2024)

Reviewer comment (Experiments):
Your framework is built on the top of RARL, which means that the adversarial action spaces are specifically chosen to be different from those of the protagonist agent in order to exploit domain knowledge. Could you explain more about how you set up the adversarial action spaces with the additional environments/tasks compared with the ones (e.g., halfcheetah, swimmer, hopper) RARL provides?

Author response:
We thank the Reviewer for this interesting question. In the spirit of the original RARL paper [1], we aimed to define action spaces which allow even a weak adversary to sufficiently disrupt the operation of the protagonist. For instance, in swingup tasks (Pendulum, Acrobot, Cartpole) we allowed the adversary to push directly on the pole(s). We found that this led to a greater robustness in trained agents, and also highlights the need for regularization as RARL is sometimes not able to solve these sensitive environments. For the specifics of the adversarial action spaces for each environment, we point to Table 3 in Appendix B.2.

Reviewer comment (Theory):
Is your proposed method also compatible with Noisy Robust Markov Decision Process (NR-MDP), which MixedNE-LD builds on top of? If so, what the role of $\alpha$ will be in NR-MDP? Is it just similar to the concept of $\delta$ mentioned in MixedNE-LD paper for defining the limit of the adversary?

Author response:
Thanks for raising the interesting NR-MDP framework. QARL should be compatible with the NR-MDP framework by running our curriculum method over the rationality of the adversarial mixing agent in NR-MDP while keeping the action-mixing ratio fixed. We posit that this would be more successful than running the curriculum over the action-mixing ratio directly ($\delta$ in MixedNE-LD) as our experiments suggest that our curriculum over rationality generally leads to better performance and robustness than methods which run a curriculum over adversary disturbance magnitude such as [2] and Force-Curriculum (described in Appendix A of our work).

Reviewer comment (Writing):
On page 2, a statement: "Conversely, in a setting where the protagonist is completely rational and the adversary is completely irrational, i.e., it plays only random actions...". Do you mean that the protagonist is playing only random actions? Then what action of the adversary take?

Author response:
We apologise for the unclear wording here. The pronoun 'it' in 'it plays only random actions...' refers to the adversary. We have updated the paper to make this more clear.

Reviewer comment (Reproducibility):
Could you provide the hyperparameter tuning or how you select $\xi$ and $\epsilon$? I think they will also influence the adversarial learning process.

Author response:
The selection of $\xi$ can be done considering the environment at hand, and the exact values used are listed in Table 3 in Appendix B.2. We find that choosing a relatively low value (as a fraction of the total achievable score for each environment) is the most reliable method for tuning $\xi$, as this progresses the curriculum under the condition that the protagonist has not catastrophically failed at solving the task at each iteration. We thank the reviewer for raising this question. We have included a new ablation study on $\xi$ in Figure 17 of Appendix C.2 which further demonstrates our reasoning.  

The hyperparameter $\epsilon$ is tuned along with the parameters of the initial and target temperature distributions. Initial tuning of these hyperparameters resulted in a stable set of values that worked consistently well across all the environments tested. This is thanks in part to the relationship between temperature and agent rationality remaining fairly consistent across different adversarial action spaces. The value for $\epsilon$ and other curriculum hyperparameters are listed in Table 4 of Appendix B.3.

Reviewer comment (Reproducibility):
Could you please point out which script.py under mujoco_experiments is your proposed method?

Author response:
The script that describes our method is:  

`code/src/experiments/mujoco_experiments/qarl_experiment.py`

Reviewer comment (Presentation):
Once the temperature had been split between the two players ($\alpha$ and $\beta$), I found the connection to QRE a bit tenuous. This is okay, but I think it would be better to present the QRE with one temperature and then state that you find better performance by using two.

Author response:
We thank the reviewer for raising this point. We refer to Chapter 4 of [1], where heterogenous QRE theory is discussed. As our aim is to robustify the protagonist agent, we choose not to excessively bound the rationality of the protagonist so that it can reasonably solve the task at each stage and efficiently progress to the end of the curriculum, where the robustness benefits are maximized. Nevertheless, we find the suggestion of experimenting with homogenous QRE to be an interesting venture, and we have included it in a new ablation study in Appendix C.2 (Figure 18), where we compare homogeneous QRE against heterogeneous QRE in QARL. The results confirm that the use of heterogeneous QRE enables to achieve superior performance.

Reviewer comment (Evaluation):
I would have liked to see results over the course of training. Do you see monotonic improvement? How challenging is the saddle point problem (adversary vs protagonist) experimentally? Can you plot $(\alpha, \beta)$ over training?

Author response:
Despite QARL relying on entropy-regularization and, thus, not having guarantees of monotonic improvement, the empirical performance of the protagonist usually increases steadily thanks to the curriculum, which halts if the agent is no longer able to solve the task. For a typical example of what training looks like, we refer the reviewer to Figure 6a of Appendix A which plots the return of the protagonist agent and the agent temperatures at each iteration. Please note that the adversary return will simply be the negative of the protagonist return, and the protagonist temperature is tuned using the automatic temperature curriculum of SAC and is therefore independent of the curriculum.

Reviewer comment (Writing):
Note McKelvey and Palfrey defined the QRE along with a more specific QRE, called the limiting logit equilibrium, that is obtained by annealing the temperature from infinity to zero (homotopy approach).

Author response:
Thank you, this is definitely correct. QARL utilises the same logic as the homotopy approach mentioned above, as described in Section 4.2 of our work.  

We agree with the Reviewer. Our aim in mentioning the homotopy method was to highlight a conceptual similarity of our method with this well-established optimization technique. However, we have chosen to remove this point from our paper as we would rather not mislead a reader or distract from our method.

Reviewer comment (Theory):
Equation 4: You describe this as a maximum entropy formulation, but this looks like an entropy regularized approach rather than selecting the equilibrium with maximum entropy (which is different).

Author response:
We understand this concern. However, we have chosen to use these terms interchangeably in the same manner as [1,2,3]. We can change it to "entropy regularization" if the Reviewer will consider it important.

Reviewer comment (Writing):
Irrational: I understand why you've chosen to pair "irrational" against "rational", but I think it's inaccurate. I think you mean "random". Note that a random policy is not necessarily irrational (e.g., random is Nash in rock-paper-scissors).

Author response:
Thank you. In the revised version, we have corrected all mentions of 'irrational' behavior in our work with 'minimal rationality' or 'random'.

Reviewer comment (Theory):
Section 4.1: You say "In Markov games... QRE can be computed in closed form...". This is not true. If we could compute QREs in closed-form (at any temperature), we could compute nearly exact Nash equilibria in closed-form. I think you just mean that computing the denominator of equation 5 is difficult due to the integral. The integral becomes a sum with finite actions, but you still have to solve a fixed-point problem to compute a QRE.

Author response:
We agree that this statement is too broad, and we have updated it in the revised version.

Reviewer comment (Theory):
Why do you define a probability distribution over $\alpha$ instead of just controlling $\alpha$ point-wise?

Author response:
We point the Reviewer to the ablation study in Figure 3 investigating other curriculum methods. **Point Curriculum** refers to a curriculum of point-wise values of $\alpha$ rather than a distribution. We find that our method (**Auto**) results in the greatest degree of robustness. The idea is that by sampling over several similar values of temperature, both the adversary and protagonist agent are better able to generalize their behaviour at each iteration of the algorithm, smoothing the progression of the curriculum.

Reviewer comment (Theory):
Equation 11: Do you minimize this for a fixed $\lambda$ and $\eta$?

Author response:
The $\lambda$ and $\eta$ in Eq. 11 are the Lagrange multipliers and are not set by us; rather, the minimization described in (11) (which is equivalent to that described in (10)) is performed using a minimizer function from a scientific Python library (`scipy.optimize.minimize`), which automatically adjusts the Lagrange multipliers.  

Using `scipy.optimize.minimize` with the 'trust-constr' method solves a Lagrangian, as described here: https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html#optimize-minimize-trustconstr.

We explicitly show this Lagrangian in (11) and use the minimizer to solve it for us. We kindly refer the Reviewer to the function `update_temperature_distribution` in `code/src/teachers/self_paced_gamma_teacher_fixed_beta.py` in our supplied code to see how the minimizer is called during the curriculum. In practice, the objective function and constraints listed in (10) are supplied directly to `scipy.optimize.minimize`.

Reviewer comment (Theory):
Do you have an easily-accesible reference for QREs with different temperature values per player (i.e., heterogeneous QREs)? I'm familiar with QREs, but never seen this and I couldn't track down the precise Goeree reference you cite. To my knowledge, this is a deviation from the QRE concept, but still makes for an interesting story as inspiration for your approach.

Author response:
Thank you for bringing up this point. We point the Reviewer to Chapter 4 of [4], which describes the mathematical and implementation details of heterogeneous QRE and its deviations from homogeneous QRE.  

Thank you for sharing this additional reference. We acknowledge that an open-source reference for heterogeneous QREs is important to the clarity of our paper, and we included it as an additional reference in Section 4 of our work.

Reviewer comment (Theory):
Relies on heuristic approach to design curriculum schedule with no theoretical guarantee on convergence behavior.

Author response:
We thank the Reviewer for raising this point. We acknowledge that as both adversarial and curriculum methods in deep RL are typically lacking in convergence guarantees, it is prohibitive to prove well-defined behaviour of our method in the theoretical sense. We do find that the heuristic curriculum parameters outlined in Table 4 of the Appendix are relatively easy to find during initial tuning of the algorithm and are broadly applicable to a wide range of environments, as evidenced by the improved empirical convergence properties of our algorithm compared to the tested baselines, as seen in Section 5.

Reviewer comment (Writing):
Where is policy $\pi$ used in the equation 6,7,8? What is variable $\mathcal P$ as the conditioning parameter?

Author response:
In equations 6,7,8 the optimal soft value function $V^*$ and optimal soft action-value function $Q^*$ are those obtained when following the optimal temperature-conditioned policy $\pi^*$.

- $\mathcal{P}$ is the transition probability density function, as defined in the Preliminaries Section 3.

- $Q^*$ and $\pi^*$ are dependent on each other. $Q^*$ is the fixed point of the soft Bellman backup operator, and $\pi^*$ is the optimal soft policy that results from soft policy iteration to convergence. This formulation is typical for maximum entropy RL [1,2].

Reviewer comment (Theory):
Is there any insight into why the curriculum approach helps optimization compared to direct adversarial training?

Author response:
Thank you for raising this point. It is known [3,4] that gradient-based algorithms in non-cooperative games are not always guaranteed to converge (e.g., due to bad initializations or a highly non-concave non-convex objective landscape). Entropy regularization is one method by which a difficult objective landscape can be made 'easier' to optimise over. In the single-player case, augmenting the objective with an entropy term relaxes the fluctations in a difficult objective landscape, facilitates the use of a higher learning rate, and reduces the number of local optima in the objective. These claims are studied qualitatively in [5], which sums up the benefits of MaxEnt RL as, 'The smoothing effect of entropy regularization, if decayed over the optimization procedure, is akin to techniques that start optimizing, easier, highly smoothed objectives and then progressively making them more complex'.

Reviewer comment (Experiments):
How well does the computation overhead of sampling different rationality level scale?

Author response:
The computational overhead of sampling temperatures scales linearly in the number of per-agent training episodes $N$ (Algorithm 1). In practice, this sampling contributes a trivial fraction of the computational overhead since we model the temperature using a closed-form distribution.

The total curriculum process of each iteration (including performance estimation, updating the temperature distribution, and sampling) account for $\frac{1}{60}$ of the computational overhead on average. We would like to point that this can be also verified through our submitted codebase in the supplementary material.

Reviewer comment (Theory):
To train the protagonist by QARL, this paper requires that the rationality of the adversary can be controlled. This is a strong assumption, and its motivation is not well-justified. It would be better if this paper could show the performance of QARL even if the rationality of the adversary is not tuned but just increasing. Most importantly, this weakness limits the possible application of this algorithm to scenarios where a reliable simulator is available because the control over the rationality of the adversary may only be possible in a simulator.

Author response:
We thank the Reviewer for pointing this out. We want to clarify that by leveraging the connection between entropy regularization and Quantal Response Equilibria, we show that we are able to control the rationality of the agent by simply changing the temperature parameter of the entropy regularization term. This is not different from the way regular Deep RL algorithms, e.g., SAC, tune the temperature during training, and it is independent to the environment and/or the used simulator. In fact, this can be done even when training online in real-world applications, as done in previous works [1].

Reviewer comment (Experiments):
Estimation of (12) consumes non-negligible online trajectories. The way to tune the rationality hyperparameter $\alpha$ is expensive.

Author response:
In practice, the $M$ rollouts described in Algorithm 1 required to estimate the expected performance of the protagonist agent result in negligible computational overhead when training in simulation. In one iteration of QARL (lines 2-23 of Algorithm 1), the majority of computation is used in the iterative training of the deep RL agents themselves, which is already present in the robust adversarial RL framework. Additionally, the minimization described in (10) is performed using a standard minimizer from Scipy (`scipy.optimize.minimize`) which results in $\frac{1}{60}$ of the computational overhead on average. We would like to point that this can be also verified through our submitted codebase in the supplementary material.

Reviewer comment (Experiments):
Have you tried any other heuristics for tuning $\alpha$? If so, could you briefly discuss them?

Author response:
Thanks for raising this question. We acknowledge the importance of evaluating other approaches for the tuning of the temperature. In fact, we have already provided in the original submission an ablation study on different ways to tune the temperature, as it can be seen in Figure 3. We evaluate the obtained robustness on $2$ locomotion problems for $4$ different temperature tuning approaches:
- **Auto:** Our automatic curriculum generation approach;
- **Linear Curriculum:** Linearly annealing of the temperature, i.e., linear increase of rationality;
- **Point Curriculum:** Using a scalar value to model the temperature instead of sampling it from a probability distribution;
- **Reduced Sampling:** Sampling only one value at each iteration, i.e., $N=1$.

We show that **Auto** outperforms the other approaches. This showcases that both the optimization problem (10) we propose and the use of a probability distribution to model the temperature are crucial in QARL.