Paper: Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity (ICLR-2024)

Reviewer comment (Theory):
The framework of the robust MDP and risk-sensitive MDP is not clear enough: The authors fail to explicitly differentiate between stationary and non-stationary policies, as well as between stochastic and deterministic policies. While Equations (1) and (2) suggest that the authors are considering a robust MDP with a non-stationary and deterministic policy, Equations (7) and (8) indicate a stationary and stochastic policy for the risk-sensitive MDP. A similar inconsistency is observed in the transition kernel: while the robust MDP seems to consider the worst-case transition kernel for each time horizon (as evidenced by the decision variables in the infimum problem of Equations (1) and (2)), the risk-sensitive MDP presumes the existence of a transition kernel that is independent of time \( t \).

Author response:
We sincerely apologize for the confusion. However, we would like to clarify that our definitions and derivations are technically correct. For the sake of generality, in Eq (1),(2) and (3) of the definition of robust MDP, we allow the policy as well as the transition kernel to be non-stationary. However, Theorem 2 points out that the sup-inf solution of equation (1)(2)(3) can be obtained by stationary policies and stationary transition kernels (Eq (11),(12)). Thus it is sufficient to restrict the later discussions to stationary policies and transition kernels, which is why our definition of value functions (e.g. Eq (4) and (7)) are only defined for stationary policies. Regarding the reviewers' concern about the differentiation between deterministic policies and stochastic policies, we apologize for our inaccurate notations which caused this confusion. The optimal value function is actually defined as the supremum over all possible policies which can be potentially stochastic, non-stationary and non-Markovian. However, the optimal solution can be obtained by a deterministic stationary Markov policy. We have updated our definition of equation (1)(2)(3) and added the above discussion as a footnote in the revised paper (highlighted in blue).

We would also like to note that our definition and analysis resemble the standard argument for risk-neutral MDP counterparts, where the original definition of the optimal value function considers taking the supremum over all possible non-stationary, even non-Markovian policies, yet it is shown that the optimal policy can be obtained by a stationary deterministic Markov policy. We hope that the above discussion clarifies the reviewer's confusion and we are open to further discussions if there are any follow-up questions from the reviewer.

Reviewer comment (Theory):
Within the framework of risk-sensitive MDP, the value function for a given policy \( \pi \), denoted as \( \tilde{V}^{\pi}(s) \), is defined by equation (7). The optimal value function, \( \tilde{V}^\star \), is defined as \( \max_{\pi} \tilde{V}^{\pi} \). Could the authors elucidate how \( \tilde{V}^\star \) satisfies the equation (9)?

Author response:
Thank you for reading carefully into the technical details in the paper! We would like to clarify that the definition of the optimal value function $\widetilde{V}^\star$ is actually initially defined as Eq (9) and then in Theorem 2, we show that $\widetilde{V}^\star(s) = \max_{\pi} \widetilde{V}^\pi(s)$, i.e. the optimal value function can be achieved by a Markov policy, which is proved in Appendix E at the beginning of page 24, where it shows that there exists a $\pi^\star$, such that $\widetilde{V}^\star = \widetilde{V}^{\pi^\star}$.

Reviewer comment (Theory):
As noted in remark 7, the sample complexity of the proposed algorithm appears less favorable than that detailed in reference [11], assuming that there is no function approximation error in authors' approach, \( \epsilon_c = 0 \). What then are the advantages of the corresponding RFZI algorithm? Furthermore, given the resemblance between Equations (7) and (9) and the traditional Bellman equation, and considering that the operators are proven contractions, is it feasible for the authors to employ value iteration or policy iteration to resolve the MDP? Such methods should theoretically exhibit rapid (geometric) convergence and remain unaffected by unknown exact gradients present in policy gradient methods.

Author response:
Regarding the reviewer's question about the advantage of RFZI over reference [11], we would like to clarify that reference [11] considers the standard non-robust/risk-neutral RL, which is a different and possibly simpler setting compared with the robust/risk-sensitive setting considered in this paper. Based on the suggestion by other reviewers, we also added a discussion that compares our paper with an offline learning algorithm for RMDP [61] which is a more fair comparison (see Remark 6 in the revised paper (previously Remark 7 in the original version) and the response to Reviewer SuJM [Performance comparison with RFQI algorithm in [61]] for more details).

Regarding the reviewer's question of employing value iteration or policy iteration to resolve the MDP, our RFZI algorithm actually resembles policy iteration. In specific, the motivation of the algorithm is to solve the optimal Q function by solving the Bellman equation $Q^\star =T_{Q} Q^\star$. However, $T_{Q}$ involves a term $\log E_{s'\sim P_{s,a}}$ which is hard to approximate with empirical estimation. Thus, instead of directly solving $Q^*$ using $Q^\star=T_Q Q^\star$, we introduce the Z-functions as auxiliary variables and solve the approximated fix point equation for $Z$. And the contraction of the Bellman operator is also reflected in our theoretical bound, where the first part $\frac{2\gamma^K}{(1-\gamma)^2}$ captures the effect of $\gamma$-contraction of the Bellman operators.

Reviewer comment (Theory):
At the end of remark 7, the authors claim that the choice of \( \beta \) should not be too large or too small, and would be better if it is on the same scale with \( 1-\gamma \). \( \beta \) is the radius of the ambiguity set related to transition kernel and \( \gamma \) is the discounted factor. It seems unconventional to determine the ambiguity set's radius based solely on the discount factor. Could this choice be elaborated upon?

Author response:
Thank you for the valuable question. We would first like to clarify that $\beta$ serves as an indicator of the penalty strength of the deviation of $\widehat{P}$ from $P$ instead of the radius of the ambiguity set. The smaller $\beta$
is, the larger the penalty strength. Thus it can be expected that the sample complexity bound depends on both $\beta$ and $\gamma$, as they together determine the landscape of the problem. We concur with the reviewer that it looks counter-intuitive at first glance that the choice of $\beta$ should be 'on the same scale with $1-\gamma$'. The main reason lies in the exponential dependency term $O\left(e^{\frac{\beta}{1-\gamma}}\right)$ that shows up in the upper bound. It is still unclear to us whether it is a proof artifact or intrinsic in our setting, however, there are results under similar settings that suggest this exponential dependency is fundamental (e.g. Theorem 3 in [23]). Numerical evidence also suggests that large $\beta$ would affect the learning stability and typically people choose the hyper-parameter $\beta$ to be roughly on the same scale as $1-\gamma$ (e.g. [55]). Additionally, we emphasize that the choice of $\beta$ doesn't depend solely on $\gamma$ and might be related to the specific structure of the MDP, hence validation and hyperparameter tuning is still necessary for choosing $\beta$.  We have added part of the above discussion in Remark 6 in the revised paper (previously Remark 7 in the original version).

Reviewer comment (Theory):
In 'E.2 Proof of theorem 2 (infinite horizon case)' on page 23, the authors claim that it is not hard to show \( \lim_{h\to +\infty} \bar{V}^\star_{0:h} = \bar{V}^\star, \lim_{h\to +\infty} \bar{V}^{\pi}_{0:h} = \bar{V}^{\pi} \). This assertion doesn't immediately resonate with me. Would it be possible to furnish a detailed proof for this claim? Subsequent statements, \( \bar{V}^\star = \mathcal{T}^\star \bar{V}^\star \) and \( \bar{V}^\pi = \mathcal{T}^\pi \bar{V}^\star \) also require clarification.

Author response:
Thank you again for reading carefully into the proofs! We have added a more detailed proof in the corresponding locations (highlighted in blue). The key step is to show that $|\overline{V}^\pi(s) - \overline{V}^\pi_{0:h}(s)| |\overline{V}^\star(s) - \overline{V}_{0:h}^\star(s)| \le \frac{\gamma^h}{1-\gamma}$. And then by leveraging the continuous property of the operators $\mathcal{T}^\pi, \mathcal{T}^\star$, we get $\overline{V}^\star = \mathcal{T}^\star\overline{V}^\star, \overline{V}^\pi = \mathcal{T}^\pi \overline{V}^\pi $.

Once again, we extend our sincere gratitude to the reviewer for the insightful comments and questions. We hope that our response clarifies the reviewer's concerns and we are more than willing to engage in further discussion to address any additional questions or concerns the reviewer may have.

Reviewer comment (Theory):
Could the authors provide some examples where theorem 2 is applied to establish the equivalence between some popular regularized RMDPs and risk-sensitve MDPs?

Author response:
*'Could the authors provide some examples where theorem 2 is applied to establish the equivalence between some popular regularized RMDPs and risk-sensitve MDPs?'*

One example of the soft RMDPs and risk-sensitive MDPs is the soft-KL RMDP, which is equivalent to the risk-sensitive MDP with entropy regularizer. This example is discussed in more detail in Section 5 of the paper.

Another example can be the risk-sensitive MDP with Conditional value at risk (CVaR, see Example 2 in the original version). Leveraging Theorem 2, it is equivalent to the RMDP with uncertainty set $\frac{\widehat{P}(s'|s,a)}{P(s'|s,a)}\le {\alpha^{-1}}, \forall ~s',s\in\mathcal{S}, a\in\mathcal{A} $

Reviewer comment (Writing):
In section 2, the authors introduce convex risk measures, where there is only one input in the measure $\sigma$ in (5) and (6), while there are two in examples 1 and 2. Could the authors explain what is the difference between these two $\sigma$'s?

Author response:
*'In section 2, the authors introduce convex risk measures, where there is only one input in the measure $\sigma$ in (5) and (6), while there are two in examples 1 and 2. Could the authors explain what is the difference between these two $\sigma$'s?'*

Thank you for the question! In the paper, we provide an explanation after Theorem 1 that clarifies the two inputs in the risk measure $\sigma$ as well as the penalty function $D$, which we quote as follows: 'In most cases, the convex risk measure $\sigma(V)$ can be interpreted as the risk associated with a random variable that takes on values $V(s)$ where $s$ is drawn from some distribution $s\sim\mu$. Consequently, most commonly used risk measures are typically associated with an underlying probability distribution $\mu\in\Delta^{|\mathcal S|}$ (e.g., Examples 1 and 2). This paper focuses on this type of risk measures and thus we use $\sigma(\mu,\cdot)$ to denote the risk measure, where the additional variable $\mu$ indicates the associated probability distribution.  
Correspondingly, we denote the penalty term $D(\hat{\mu})$ of $\sigma(\mu,\cdot)$ in the dual representation theorem as $D(\hat{\mu},\mu)$.' We hope that this answers the reviewer's question and we are happy to further discuss if there are any parts that remain ambiguous.

Reviewer comment (Writing):
What is "direct parameterization"?

Author response:
*'What is "direct parameterization"?'* and *'About the feasible region $\mathcal X$ of $\theta$ : why it is of dimension $SA$ 
? And why each subvector $\theta_s$ should be in the probability simplex?'*

Direct parameterization is one of the most commonly considered approaches to parameterize a policy in theoretical RL (c.f. [2]). As we have stated before Corollary 1, 'for direct parameterization, the parameter $\theta_{s,a}$ directly represents the probability of choosing action $a$ at state $s$, i.e., $\theta_{s,a} = \pi_\theta(a|s)$.' We have also added some standard references for direct parameterization in the same paragraph. 

Given this definition of direct parameterization, $\theta$ is of dimension $SA$ because for each $s$ and $a$, there's a corresponding entry $\theta_{s,a}$ that represents $\theta_{s,a} = \pi_\theta(a|s)$. Further, the subvect $\theta_{s}$ represents $\theta_s = \pi_\theta(\cdot|s)$ which lies in the probability simplex. We hope that the above response answers the reviewer's question!

Reviewer comment (Theory):
Is the right-hand side of (15) independent of the initial distribution $\rho$?

Author response:
*'Is the right-hand side of (15) independent of the initial distribution $\rho$?'*

Thank you for reading into the technical details. The right-hand side of (15) is actually dependent on the initial distribution $\rho$ because the discounted state visitation distribution $d^{\pi_\theta, \widehat P^\theta}$ (defined in Section 2) is dependent on $\rho$.

Reviewer comment (Theory):
About the feasible region $\mathcal{X}$ of $\theta$: why it is of dimension $SA$? And why each subvector $\theta_s$ should be in the probability simplex?

Author response:
*'What is "direct parameterization"?'* and *'About the feasible region $\mathcal X$ of $\theta$ : why it is of dimension $SA$ 
? And why each subvector $\theta_s$ should be in the probability simplex?'*

Direct parameterization is one of the most commonly considered approaches to parameterize a policy in theoretical RL (c.f. [2]). As we have stated before Corollary 1, 'for direct parameterization, the parameter $\theta_{s,a}$ directly represents the probability of choosing action $a$ at state $s$, i.e., $\theta_{s,a} = \pi_\theta(a|s)$.' We have also added some standard references for direct parameterization in the same paragraph. 

Given this definition of direct parameterization, $\theta$ is of dimension $SA$ because for each $s$ and $a$, there's a corresponding entry $\theta_{s,a}$ that represents $\theta_{s,a} = \pi_\theta(a|s)$. Further, the subvect $\theta_{s}$ represents $\theta_s = \pi_\theta(\cdot|s)$ which lies in the probability simplex. We hope that the above response answers the reviewer's question!

Reviewer comment (Writing):
Assumption 1, typo in the first sentence.

Author response:
*'Assumption 1, typo in the first sentence.'* 

Thank you for pointing out the typo! We have corrected it in the revised paper.

Reviewer comment (Writing):
Assumption 3, what is the norm with subscript "$2,\mu$"?

Author response:
*'Assumption 3, what is the norm with subscript "$2,\mu$"'?*

Thank you very much for the question! We recognized that we didn't give a formal definition of $||\cdot||_{2,\mu}$. We have added the definition to Section 2 in the revised paper, which is stated as below: for any function $f:\mathcal{S}\times\mathcal{A}\to \mathbb{R}$, state-action distribution $\mu\in\Delta(\mathcal{S}\times\mathcal{A})$, the $\mu$-weighted 2-norm of $f$ is defined as 

$||f||{2,\mu}  =(E_{s,a\sim\mu}f(s, a)^2)^{1/2}$. 

We have added the above definition to the preliminary section of the paper (highlighted in blue). We sincerely appreciate your contribution in highlighting this particular omission.

Reviewer comment (Novelty):
Many of the results appear to follow closely from existing work. Theorem 1 comes from [27]. The equivalence between soft robust optimization and convex risk measure is already established in Ben-Tal et al. (2010). It is therefore no surprise that the result in Theorem 2 would hold. Theorem 3, Lemma 2, and Theorem 4 extend the result of [85] from robust MDP to the slightly more general regularized/soft robust MDP. Section 5 proposes an algorithm that resembles the approach presented in [24] although I am not aware of a sample complexity result for that class of algorithm.

Author response:
**[Novelty of Results]** The reviewer pointed out that some of the results in this paper follow closely from existing work. Here we respectfully reference the comments from the reviewer and aim to address these observations by clarifying our contributions and novelty in comparison to prior studies.

- *'Theorem 1 comes from [27]. The equivalence between soft robust optimization and convex risk measure is already established in Ben-Tal et al. (2010). It is therefore no surprise that the result in Theorem 2 would hold.'* 

We would like to first clarify that the dual representation theorem (Theorem 1) is not claimed to be a contribution of our work; instead, we reference this theorem from classical literature, as it forms the foundation for the subsequent content of our paper. Regarding the significance of Theorem 2, we agree with the reviewer that the proof intuition follows from Theorem 1 and it might not be surprising for readers who are familiar with both dynamic programming and risk measures. However, we would like to note that it is not a direct or trivial generalization of Theorem 1, as it considers the MDP setting, which has a sequential structure that is more complicated than the static setting considered in Theorem 1. The proof (detailed in Appendix E) also requires a careful combination of dynamic programming with the dual representation theorem. As far as we know, although there are many works that discuss the relationship between robustness and risk-sensitivity (see Appendix A in our paper for more detail), there's no result that explicitly states the exact equivalence between (soft) RMPD and risk-sensitive MDP. We believe that Theorem 2, therefore, remains a distinctive and interesting contribution with its own merits.

- *'Theorem 3, Lemma 2, and Theorem 4 extend the result of [85] from robust MDP to the slightly more general regularized/soft robust MDP.'* 

Thank you for the comment and for directing us to the most related literature [85]. Here we briefly list the major differences between our paper and [85]. Firstly, we obtain an explicit analytical form of the policy gradient in Theorem 3, whereas in [85], they consider the Moreau envelope of the value function and it is implicitly computed through a double-loop manner. This immediately leads to the next two differences--different algorithm designs and different theoretical bounds. [85] needs to consider a modified policy gradient with a double-loop, whereas our algorithm removes the double-loop and studies policy gradient without modification. Given the simpler algorithm design, our iteration complexity also has a better dependency on the size of the state and action spaces $|\mathcal{S}|,|\mathcal{A}|$ as well as the distributional shift factor $M$ (denoted as $D$ in [85]).
However, we also acknowledge that [85] has considered settings more general than our paper, e.g. it also takes the $s$-rectangularity setting into consideration. It would be an interesting next step to see if we could combine insights from both works to have more general and improved results.

- *'Section 5 proposes an algorithm that resembles the approach presented in [24] although I am not aware of a sample complexity result for that class of algorithm.'* 

We express our gratitude to the reviewer for recognizing the novelty of our sample complexity result. We would like to first clarify that the objective function considered in [24], namely the exponential utility, differs from our objective function (see more elaboration in the third paragraph in 'Other related works' in Appendix A). Additionally, the learning settings as well as the algorithms are different. [24] considers online learning in the tabular setting and the algorithm is based on Q-learning, whereas we consider offline learning with function approximation and the algorithm resembles policy iteration type of algorithms.

Reviewer comment (Writing):
The concept of regularized robust is misleading. In problem (2) it is the adversarial transition probability matrix that is being regularized. This is completely different from regularizing the policy as is usually understood as "regularized MDP". I would suggest calling it differently to avoid this confusion. Namely, in the robust optimization literature, the concept of regularizing the adversaries’ actions is referred to as soft-robust, comprehensive robust, or globalized robust optimization (see references below).

Author response:
**[Term of 'regularized robust MDP']** The reviewer pointed out that the term 'regularized robust MDP' might be misleading. We sincerely thank the reviewer for presenting more accurate alternatives and providing related references. After reading the references, we fully agree with the reviewer that 'soft robust MDP' indeed encapsulates the intended concept more accurately. We have changed the name of the term to `soft RMDP' and added related references in our revised paper. We will use soft RMDP in the response as well.

Reviewer comment (Writing):
Page 5, in the middle "Risk-Sensitive MDPs" misspells "Senstive".

Author response:
We would like to once again thank the reviewer for a thorough reading of our work! We have fixed the typos and changed the notation based on the reviewer's suggestions.

Reviewer comment (Writing):
Page 5, the line above equation (7), misspells "valule".

Author response:
We would like to once again thank the reviewer for a thorough reading of our work! We have fixed the typos and changed the notation based on the reviewer's suggestions.

Reviewer comment (Writing):
Page 25, paragraph under section C, last part, repeats the expression "is defined as is defined as".

Author response:
We would like to once again thank the reviewer for a thorough reading of our work! We have fixed the typos and changed the notation based on the reviewer's suggestions.

Reviewer comment (Writing):
Page 26, the notation would be clearer if the two “T^\pi” and “T^*” operators were denoted as “bar T^\pi” and “bar T^*” since they are defined in terms of the robust MDP.

Author response:
**[Typos/clarifications]** We sincerely thank the reviewer for carefully going through the paper and pointing out typos and other possible corrections and improvements. We have incorporated the reviewer's suggestions into the revised version of the paper. (For the notation $\mathcal{T}^\star, \mathcal{T}^\pi$, because it is originally defined as the Bellman operator for the risk-sensitive MDP instead of soft RMDP, we decide to keep the original notation $\mathcal{T}$ rather than $\overline{\mathcal{T}}$ as suggested by the reviewer.) It seems that the page numbers referenced by the reviewer might be slightly inaccurate. We remain open to making additional revisions if our current modifications do not align with the reviewer's intended recommendations.

Once again, we are grateful for the reviewer's insightful comments and constructive suggestions! We remain at the reviewer's disposal for any further inquiries or points requiring additional discussion. Your valuable input is greatly appreciated.

Reviewer comment (Writing):
Page 26, last part of Proof of Lemma 1, "Lemma 1 is an immediate corollary of Lemma 8" should be "Lemma 3". Also, "\tilde V and \tilde V^\star in (4) and (3)" should be "(7) and (9)".

Author response:
We would like to once again thank the reviewer for a thorough reading of our work! We have fixed the typos and changed the notation based on the reviewer's suggestions.

Reviewer comment (Writing):
Page 27, in Lemma 4, the definition of \mathcal{T}^* should be clarified.

Author response:
We would like to once again thank the reviewer for a thorough reading of our work! We have fixed the typos and changed the notation based on the reviewer's suggestions.

Reviewer comment (Theory):
It is frustrating to see many references in the bibliography with no discussion of them in the text body. Specifically, how does the regularization term of this work compare to [18]? Also, the regularized+robust objective considered in this work is reminiscent of [32], who also combine the two to derive a convex formulation of RMDPs. This raises the question: does the same convex formulation apply to the authors' regularized RMDPs? Is it the aforementioned convex property that ensures gradient domination in this setting as well?

Author response:
**[Comparison with [18] and [32]]** The reviewer raised the question of how our regularized MDP compares with the regularization term in [18]. We would like to first apologize for our misleading use of the term 'regularized robust MDP' (which is also pointed out by the third comment in the 'weakness' section of  Reviewer RSEz). As is also pointed out by Reviewer RSEz, the regularization in [18] and ours are different in the sense that the 'regularized MDP' in [18] considers regularization on the policy whereas in our setting the regularization is on the transition probability matrices. To avoid confusion, as suggested by Reviewer RSEz, we have changed 'regularized robust MDP' to 'soft robust MDP' in the revised version of the paper. Thank you for raising this valuable question! 

The reviewer also asked whether the same convex formulation in [32] applies to the regularized RMDPs (soft RMDPs). We are also interested in the same question and we actually had a brief discussion with one of the authors in [32]. Based on the discussion and our preliminary attempt, it might not be a trivial combination, and it is an interesting future direction to study how to leverage our formulation to get simplified convex formulations for (soft) RMDPs. In response to the reviewer's question of whether the gradient domination property is ensured by the above convex formulation, our current proof leverages the performance difference lemma and the key technique is inspired by the proof of gradient domination in standard MDP [2]. It would be an interesting research topic to study if the gradient domination property can be proved from the convex formulation perspective, but currently, it is unclear to us whether the convex formulation is directly related to the gradient domination property.

Reviewer comment (Novelty):
As mentioned in Rmk 4, sampling from $\hat{P}^{\theta}$ rather than the nominal represents the main challenge in estimating the policy-gradient. The work [101] explicitly formulates the policy gradient in RMDPs: by focusing on $\ell_p$-ball constrained uncertainty sets, the authors are able to explicitly derive the worst transition model and, as a result, a robust policy-gradient that is more amenable to be estimated from nominal samples. Specific assumptions on the convex risk measure $\sigma$ could similarly be leveraged to make explicit a risk-sensitive gradient. For example, departing from the equivalence between coherent risk measures and RMDPs with a risk envelope, one may write a robust policy-gradient as in [101] and broaden the class of measures to be considered while preserving tractability of the gradient.

Author response:
**[Application of techniques in [101]]** The reviewer asked whether we could apply similar techniques in [101] (which we believe is the same reference as [44] in the original paper, we have updated the publisher info of [44] in the revised paper) by explicitly deriving the worst transition model $\widehat{P}^\pi$. We agree with the reviewer that in certain cases the worst-case probability transition model can indeed obtain an explicit/analytical form that can simplify the problem to a certain extent (e.g. entropy risk measure $\widehat{P}^\pi$ takes the form $\widehat{P}^\pi(s'|s,a)\propto P(s'|s,a)\exp(-\beta V^\pi(s'))$). We believe that in this case we could leverage techniques such as importance sampling or rejection sampling to make gradient estimation more amenable. However, based on our own calculation, the sample complexity analysis is more technically involved, which is one of the main reasons why we considered the RFZI algorithm instead of policy gradient algorithm for the sample-based learning setting and we left it as the next step of our research to study the performance of sample-based policy gradient for risk-sensitive MDPs and (soft) RMDPs.

Reviewer comment (Experiments):
The offline learning part is a bit of an outlier in this paper. In particular, the motivation for Section 5 is unclear. Simulation experiments, on the other hand, would study the risk-sensitive behavior of regularized-robust optimal policies and compare them to robust optimal policies.

Author response:
**[Motivation for the offline learning algorithm RFZI]** The reviewer has raised questions regarding the motivation behind the offline learning algorithm presented in Section 5. The primary motivation behind Section 5 is to introduce a sample-based learning algorithm for risk-sensitive MDPs and soft robust MDPs. This is essential as Section 4 only focuses on the learning algorithm under full information and explores the iteration complexity. Ideally, it would be better to design sample-based policy gradient methods.  However, as mentioned in Remark 4, it is difficult to estimate the policy gradient by samples, which is why we first consider the RFZI algorithm which resembles policy iteration rather than policy gradient. The reason for considering the specific setting of offline learning algorithm with nonlinear function approximation is that, as mentioned in the introduction, it is a less-studied setting for RMDPs and we hope that our results could bring insights to make algorithm design and theoretical analysis easier for this less-studied scenario. We apologize for not explaining our motivations clearly and we hope that the above explanation clarifies the confusion of the reviewer. We have included part of the above discussion at the beginning of Section 5 and we are happy to have further discussions. We have also added more experimental results that study the robustness/risk-sensitive behavior of soft robust optimal policies and compare it with the robust optimal baseline based on the reviewer's suggestions (see Figure 4 and its related paragraphs in Appendix B).

Reviewer comment (Theory):
It is not clear how the performance gap of Thm 5 compares to that of RFQI [61]. The paper briefly mentions the computational cost incurred by introducing dual variables in [61], but the performance gap obtained from each approach should be further discussed. More generally, Section 5 raises several open questions without further discussion, which is frustrating: can Assm 2 be avoided? How to empirically select the penalty factor $\beta$ and shrink the performance gap? Is the exponential dependence a proof artifact or intrinsic to the problem?

Author response:
**[Performance comparison with the RFQI algorithm in [61] and open questions in Section 5]** The reviewer suggests adding a comparison with the sample complexity bound of RFQI algorithm in [61]. We first thank the reviewer for the valuable suggestion. We agree with the reviewer that a comparison between the results of the two algorithms is necessary and we have added related discussions to the revised paper (highlighted in blue). Here we quote the discussion: 'We also compare our performance bound with the RFQI algorithm which considers a similar offline learning setting and obtains sample complexity of $O\left(\frac{\log(|\mathcal{F}||\mathcal{G}|)}{(\beta\epsilon)^2(1-\gamma)^6}\right)$, where $\beta$ in their setting is the radius of the uncertainty set. Note that both results share the same dependency on $\epsilon$ and the concentrability constant $C$. However, the bound in [61] includes an additional term on the size of the dual variable space $\log |\mathcal{G}|$, whereas we have the exponential dependence term $e^{\frac{2\beta}{1-\gamma}}$.' Yet we would also like to note that the two settings are not completely comparable as we consider different types of uncertainty sets and the algorithms are very different.

The reviewer also raised questions on the assumption and the theoretical bounds in Section 5. We address the questions as follows:

- *'Can Assumption 2 be avoided?'* We would like to refer the reader to Remark 6 (now Remark 5 in the revised paper) where we provide a detailed discussion on Assumption 2. In particular, Assumption 2 is adapted from the
corresponding notions defined for the standard MDP setting [11], where they also demonstrate the necessity of this assumption for standard RL with function approximation. However, for the risk-sensitive and soft robust MDP settings, we currently don't have theoretical results to verify the necessity of Assumption 2. It would be an interesting
open question whether Assumption 2 is also necessary.


- *'How to empirically select the penalty factor and shrink the performance gap?'* Remark 7 (now Remark 6 in the revised paper) provides a brief discussion on the choice of $\beta$ where we quote as follows:  the term $\beta^{-1}e^{\frac{\beta}{1-\gamma}}$ first decreases and then increases with $\beta$ as it goes from $0$ to $+\infty$, suggesting that the hyperparameter $\beta$ also affects the learning difficulty of the problem. The choice of $\beta$ should be neither too large nor too small, ideally on the same scale as $1-\gamma$. We would also love to add further discussions if the above discussion is not detailed enough!

- *'Is the exponential dependence a proof artifact or is it intrinsic to the problem?'* Thanks for the question! It is still unclear to us whether it is a proof artifact or intrinsic in our setting, however, there are results under similar settings that suggest this exponential dependency on parameter $\beta$ and the effective length $\frac{1}{1-\gamma}$ is fundamental (e.g. Theorem 3 in [23]). We have added the above discussion to Remark 6 in the revised paper (previously Remark 7 in the original version, highlighted in blue).

Reviewer comment (Writing):
The second paragraph of Section 1 is confusing, particularly the distinctions between "when the model is given", "cases of unknown models... most are model-based for tabular cases", and "model-free setting". How are the shortcomings mentioned in this paragraph addressed in the present work? How do the authors distinguish between "unknown model" and "model-free"? Does it refer to the uncertainty set or the nominal model? For example, [48] is model-free in the sense that it does not take the uncertainty set as given but estimates it from data. Most robust learning approaches are model-based in the sense that they consider the uncertainty set to be known and solve robust planning from there.

Author response:
**[Discussion between terms 'unknown model', 'model-based', 'model-free']** We apologize for the confusion. By 'unknown model' we refer to the setting where the nominal probability transition $P$ model is unknown. Both 'model-based' and 'model-free' methods belong to the setting with 'unknown model'. Although 'Model-based' methods don't have knowledge about the nominal probability transition, they generally keep an empirical estimate of the nominal model $P^{\textup{est}}$ and the algorithms rely heavily on the estimated model $P^{\textup{est}}$. However, in comparison 'model-free' algorithms don't require this empirical estimation step and thus might be more suitable for large-scale problems. We categorize [48] as model-based because their algorithm requires an empirical estimation (denoted as $\hat{P}_k$ in [48]) as an input. Our algorithm only requires learning the Z-function and does not rely on an estimated model and thus belongs to the 'model-free' category. We have added the above discussion as a footnote in the revised paper. Additionally, we would also like to clarify that for RMDP, people generally refer 'known/unknown model' as the nominal model rather than the uncertainty set. In the RMDP setting, the uncertainty set is generally assumed to be known and pre-given as an algorithm input. Based on our understanding of [48], we believe they still require knowledge of the uncertainty set but don't require knowledge of the nominal model.

Reviewer comment (Writing):
Rmk 1 is confusing. The meaning of "the boundary of the uncertainty set" and "hard boundary" is unclear. Referring to the fact that $(\hat{P}_t)_t$ are in the whole simplex for regularized RMDPs, as opposed to a specified uncertainty set $\mathcal{P}$ in RMDPs, would help in understanding the motivation for that setting. Also, it is unclear whether this remark is related to the discussion on "model-free" vs "model-based" in the introduction.

Author response:
**[Clarification of Remark 1]** We apologize for the imprecise language and thank the reviewer for pointing out potential improvements. We have revised the sentence based on the suggestion of the reviewer. Regarding the relationship between 'model-based' and 'model-free' learning, as mentioned in the previous paragraph, for RMDP, people generally refer 'known/unknown model' as the nominal model rather than the uncertainty set, so it is not directly related.

Reviewer comment (Theory):
Thm 1 seems to simply relate the double Fenchel conjugate with the original function. If that is the case, it should be mentioned in the text. If not, how is this result different?

Author response:
**[Relationship between Theorem 1 and Fenchel conjugate]** The reviewer's understanding is correct. In the finite-dimensional case, the penalty function $D$ is the Fenchel conjugate of $\sigma$. We have added a short remark in our revised paper to make it clearer. We would also like to clarify that Theorem 1 is not claimed to be a contribution of our work; instead, we reference this theorem from classical literature [27], as it forms the foundation for the subsequent content of our paper.

Reviewer comment (Theory):
An intuitive explanation is needed for why the value function is different from Markov risk measures of previous works when the policy is stochastic, but the same when it is deterministic.

Author response:
**[Intuitive explanation of the difference of the value function for risk-sensitive MDP and the Markov risk measure]** Thanks for the question. Due to the space limit, the more detailed explanation and comparison of the risk-sensitive MDP and the Markov risk measure was deferred in Appendix C. Intuitively speaking, the Markov risk measure also takes the risk generated by the randomness of the policy into account whereas our definition treats the randomness from the policy in a risk-neutral manner and only considers risk from the uncertainty of the transition probability. This intuitively explains why the two notions are equivalent for deterministic policies but not for stochastic policies. We have added the above intuitive explanation to Appendix C in the revised paper (highlighted in blue). We would also like to emphasize that our definition of risk-sensitive MDPs has its own benefits. It results in a cleaner equivalence with (soft) RMDPs and better convergence property compared with directly optimizing the Markov risk measure, which is known to be possibly non-gradient dominant (c.f. [36]).  Consequently, our results provide an alternative approach for the efficient optimization of the Markov risk measure, despite modeling risk in a different manner.

Reviewer comment (Writing):
Last line of Sec. 1 contains "appendix" without capital letter.

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!

Reviewer comment (Presentation):
Eqs. (12), (14), (18) are numbered without being referred to in the text.

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!

Reviewer comment (Writing):
Missing capital letters in references: Markov in [6, 10, 32, 35, 36, 43, 48, 54, 59, 71, 94] and others, Bellman in [4].

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!

Reviewer comment (Presentation):
Published/workshop papers should be referred to as such: [16, 17, 20, 21, 22, 40, 44, 52, 61, 78, 85, 96].

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!

Reviewer comment (Presentation):
Section 3 is very short. The "risk-sensitive MDPs" paragraph of Section 2 could be incorporated into Section 3.

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!

Reviewer comment (Writing):
There is a typo in Assmp 1.

Author response:
**[Suggested corrections]** We extend our sincere gratitude to the reviewer for carefully going through the paper and pointing out typos, inaccurate reference information, and other corrections! We have fixed the typos and references, deleted unnecessary content and remarks, and revised our paper based on the suggestions from the reviewer. We have also provided additional elaboration in the introduction, specifically addressing the motivation behind our offline learning algorithm and how it overcomes the shortcomings highlighted earlier. Given the richness of the field of robust and risk-sensitive learning, we regret that we could not include all the literature in the main text. We still kindly refer the reviewer to Appendix A for a more comprehensive comparison with prior works.

We would also like to note that the publisher information is not updated for a few papers in the reference because they are not officially published yet based on our literature search. We would be happy to make further updates on the references if there are any further questions or suggestions! 

With regard to the reviewer's suggestion of putting the definition of risk-sensitive MDP into Section 3, we apologize that, after careful consideration, we still decide to keep the original structure. We believe that maintaining Section 2 as a self-contained section with all the preliminaries logically aligns with the overall flow of the paper. We are open to further discussion regarding the paper's structure and appreciate the reviewer's input on this matter.


Once again, we sincerely thank the reviewer for dedicating time and effort to a thorough review of our paper. We hope that our response could clarify the questions and concerns of the reviewer and we are happy to address any other questions the reviewer has!