Paper: EWoK: Tackling Robust Markov Decision Processes via Estimating Worst Kernel (ICLR-2024)

Reviewer comment (Writing):
“The optimal policy”, “The worst-case transition probabilities”: some there are multiple optimal policies/worst-case kernels, I think that the authors should be more careful with the phrasing. I also don’t know what it means to “train” a transition kernel (last sentence of the paragraph after Eq. (7)), and I don’t know what is a “perfect value function” (paragraph before Eq. (14)). I list other inaccurate statements below.

Author response:
Thank you for pointing out the inaccuracies in our phrasing. In the updated version, we have rectified these to 'An optimal policy' and 'A worst-case kernel'. Concerning 'train a transition kernel', our intent was to refer to the process of learning a parameterized transition model. As for 'perfect robust value function', we aimed to convey that we do not have the true robust value function beforehand in practice. These imprecise statements have been addressed in the updated version.

Reviewer comment (Theory):
The Theoretical results are very weak. Nothing new in Appendix A.1 (proof of Theorem 3.2), it is already in [A].

Author response:
We agree that Theorem 3.2 is not difficult to derive. We present its proof for the sake of completeness. Our primary contribution lies in two key aspects:

* Conceptually, we show how to use Theorem 3.2 for sampling the worst kernel using the nominal kernel.
* The core theoretical contribution is Theorem 3.5, which is the heart of our algorithm. The update rule in Equation 14 is coupled, i.e., we update the transition kernel $P_{n+1}$ using the value function w.r.t. the kernel $P_n$. Hence, showing its convergence is non-trivial, necessitating a detailed and technical proof in Appendix A.1. We believe this proof does not reside in [A] but rather constitutes our core results.

Thus, we respectfully disagree with the reviewer's assessment of our theoretical contribution as weak.

Reviewer comment (Theory):
Can the EWOK be modified to cope with other uncertainty sets than KL-based uncertainty?

Author response:
Yes, we believe the high-level idea of EWoK, approximating a worst transition kernel, can be applied in other uncertainty sets than the KL-based one. The key step is to build the connection between a worst transition kernel and the nominal one, and use such connection to obtain samples approximately distributed according to the worst transition probability.

Reviewer comment (Theory):
Paragraph after Eq. (7): it is now well-recognized that the minimization problem from Eq. (7) is also an MDP, see for instance Section 3 in [1] or Section 4 in [2]. Can we use gradient-based method in this adversarial MDP to learn the worst-case transition probabilities?

Author response:
This is a good point. Thank you! We concur that the worst kernel can be computed using gradient descent, as described in Algorithm 2 in [A]. However, the convergene of Algorithm 2 to the worst kernel, has the iteration complexity of $O(\frac{S^3A}{(1-\gamma)^6\epsilon^2})$ (as guaranteed by their Theorem 4.4). In comparison, our method (update rule in our Eqn. 14) has the iteration complexity of $O(\log(\frac{1}{\epsilon}))$. We thank the reviewer for pointing out this direction and we have included the above discussion in the updated version.

[A] Qiuhao Wang,Chin Pang Ho, Marek Petrik. Policy Gradient in Robust MDPs with Global Convergence Guarantee

Reviewer comment (Writing):
Please properly introduce the parameter $\omega_{sa}$ and $\kappa_{sa}$ in Th 3.2.

Author response:
Thank you for the suggestion. We have revised our paper to make it more clear.

Reviewer comment (Theory):
Th 3.5 only states that the value converges to the robust value. It does not state that “the estimated kernel converges to the worst kernel”. Also why is \pi not indexed by the iteration, since you update it at every iteration?

Author response:
Thank you for highlighting this aspect. We note that the closeness between a kernel and a worst kernel can be measured by the difference in the value function. An example can be found in Theorem 4.4 in [A]. We call two kernels "close" if their value functions are close, because most of the time we care how a kernel minimizes the value (recall the definition of worst kernel).

Theorem 3.5 is true for all policies $\pi$ (hence all $\pi_k$). However, we can easily incorporate the changing $\pi_k$, by utilizing multi-scale algorithm reasoning. That is, running $\pi_k$ on the slower time scale and $P_k$ on the faster time scale, which allows us to decouple both updates. However, we avoided doing so, as our method is a *meta-algorithm*. It can be combined with *any* method to update the policy. This is the core message of our paper, that we simulate the worst kernel, which can be combined with any non-robust RL method to learn the robust policy.

[A] Qiuhao Wang,Chin Pang Ho, Marek Petrik. Policy Gradient in Robust MDPs with Global Convergence Guarantee

Reviewer comment (Writing):
Minor typos and notation clarity problem in the theory part (Section 3).

Author response:
Thank you for pinpointing those typos and notation issues! We have fixed them in the newest version.

Reviewer comment (Reproducibility):
About the Weakness 1 I mentioned, I would appreciate it if the authors could explain more about the empirical choice of the unknown parameter  $\omega_{s,a}$ (since the empirical average over the samples $s_i^{\prime}$ from $\bar{P}(\cdot|s,a)$ forms an overestimation of $\omega_{s,a}$ as suggested by Proposition 3.4).

Author response:
Thank you for this thoughtful question. The empirical approximation of $\omega_{sa}$ is mainly motivated by implementation considerations that we only have access to samples from the nominal transition kernel. In the future, we will explore more clever ways to approximate it to mitigate overestimation.

Reviewer comment (Writing):
Some typos and notation clarity: (i) in Eq (12) it should be $\sum_{i=1}^Nv(s_i')$; (ii) the notion of $\omega_n$ and $\kappa_n$ is not pre-defined (even with subscript $s,a$). I suggest explicitly giving them a definition (as the parameter associated with the worst case transition kernel when the target function is $v_{P_n}^{\pi}$).

Author response:
Thank you for pinpointing those typos and notation issues! We have fixed them in the newest version.

Reviewer comment (Experiments):
Another aspect that seems to be lacking is a discussion on how the radius $\beta_{sa}$ of the ambiguity set affects the algorithm's performance, although it would be transferred to new parameters $\omega_{sa}$ and $\kappa_{sa}$. I do expect that the parameter selection procedure could be discussed more.

Author response:
Thank you for highlighting this point. We would like to clarify that $\beta_{sa}$ is more of a parameter for modeling purposes. In practice, we do not set it directly. In principle, a higher $\beta_{sa}$ means a larger uncertainty set, thus a more conservative policy. However, as all non-tabular experiments never have truly rectangular or KL-constrained uncertainty sets, the influence on practical performance might vary.

Reviewer comment (Novelty):
The literature review is not comprehensive. A recent paper [1] also studied RMDPs with global optimality, and it would be helpful if the author discussed it.

Author response:
Thank you for pointing out this related work. Similar to our work, it also iteratively computes the worst kernel (appearing in its Algorithm 2). However, their method is gradient-based with iteration complexity of $O(\frac{S^3A}{(1-\gamma)^6\epsilon^2})$ (Theorem 4.4 in [1]). In comparison, our method (update rule in our Equation 14) has the iteration complexity of $O(\log(\frac{1}{\epsilon}))$ as stated in our Theorem 3.5. We have included it in the updated version.

Reviewer comment (Theory):
While we consider a practical problem lying in the KL-based $(s,a)$-rectangular ambiguity set, the only parameter that the agent can choose is $\beta_{sa}$; however, the other two parameters $\omega_{sa}$ and $\kappa_{sa}$ would be settled directly. Could you explain more about the relationship between $\beta_{sa}$ and the other two parameters $\omega_{sa}$ and $\kappa_{sa}$, or how can the agent reach the latter when setting the former?

Author response:
Thank you for raising the question. We apologize for the unclarity in our presentation. $\beta_{sa}$ is more of a parameter for modeling purposes, not something chosen by the agent. The connection between $\beta_{sa}$ and the other parameters $\omega_{sa}$ and $\kappa_{sa}$ are described in Eqn.(10) and Proposition 3.3. $\omega_{sa}$ and $\kappa_{sa}$ are the solutions of Eqn.(10), though we can't obtain a closed form for them. In our experiments, we consider $\kappa_{sa}$ as a hyperparameter, and setting it essentially decides a $\beta_{sa}$ for modeling the problem.

Reviewer comment (Theory):
I am not convinced yet that EWoK can be applied in a realistic domain as claimed. As the paper indicates in the conclusion section, EWoK assumes the environment is able to sample from the same state and action pair multiple times. This requirement is easy to achieve when using simulators or when there exists a perfect model, but is unrealistic in real environments. In the real world, it is almost impossible to reset the environment to the previous state and apply the same action multiple times. Given that the paper defines EWoK as an online method for realistic domains, I think this assumption contradicts the scenario which EWoK is supposed to work with. It might be more accurate if the paper reduces the scope to high-dimensional domains.

Author response:
Thank you for the suggestion. While EWoK shows potential for integration with learned models to get rid of the assumption, we concur that it is more accurate to reduce the current scope to high-dimensional domains. We have revised the paper to embody this change.

Reviewer comment (Experiments):
It might be worth checking a more difficult experiment setting, such as the one single trajectory case.

Author response:
Thank you for the suggestion. Could you provide more details about this specific setting? We would gladly conduct the requested experiments.