Paper: Provably Efficient CVaR RL in Low-rank MDPs (ICLR-2024)

Reviewer comment (Novelty):
My main concern is that the algorithm 1 is quite close to [1]. It seems that algorithm 1 is just an application of REP-UCB in Augmented MDPs. The analysis of regret bound is also similar to [1], with only difference to convert CVaR into value function (as Eq. (13) in Page 17), which makes the result not so surprising and significant. It is likely that I miss some novel analysis techniques, so please bring it out and I think it is also important to stress the novelty in the paper.

Author response:
**novel analysis techniques**: Thanks for the inquiry. Indeed, we acknowledge that our research is built on the foundational work of Wang et al., 2023, and Uehara et al., 2021. While we have taken insights from these significant studies, our work also contributes new techniques that we feel are substantial advancements in the field.

* Firstly, our algorithm achieves efficient exploration while performing representation learning, therefore answering the open problem raised by Wang et al., 2023. Technically, such a tradeoff is hard because we need to carefully maintain two time-dependent datasets (because we study the finite-horizon episodic setting) that provide a diversified exploration. Please see more details about the sampling in our Response to Reviewer htXD. 
* Secondly, our discretized planning oracle, CVaR-LSVI, requires new techniques.
First, we need to show that LSVI works for CVaR with continuous reward. This is not trivial because the augmented MDP involves the evolution of the budget, which can be non-linear.  Second, we also need to establish a new concentration inequality for CVaR-LSVI (lemma D.1) because we introduce the new budget variable in the regression and need to compute its covering number.
We believe this oracle's design can benefit the whole risk-averse RL community.

Considering the points mentioned, we maintain that our contributions are meaningful and substantially enrich the current research landscape. In the updated manuscript, we will more strongly emphasize these contributions. Please also refer to our response to Reviewer i64s for more details. We are more than happy to clarify any concerns.

Reviewer comment (Theory):
The authors claim the worse dependency of regret bound on $H^7$ is due to the non-stationary transitions in Page 7. I want to point out it may be wrong. Another previous work considers regular low-rank MDPs with finite horizons and time-dependent transition kernels [2], which is the same as this paper, even has better dependence on $H^3$. It is likely that the worse dependency on $H$ of this paper is due that [1,2] assume that the accumulative discounted reward is in $[0,1]$ and this paper only assumes $[0,H]$. I guess that applying similar assumptions and techniques to clip value functions in [2], the regret bound may acheive the same dependence on $H^3$.

Author response:
**Dependency on $H^7$**: We agree with the reviewer that if we use the $[0,1]$ reward assumption and the corresponding clipping, our bound can be improved to $H^5$. Besides, We think it might be possible to reduce another $H^2$ by utilizing a Bernstein-type bonus in the algorithm instead of the Hoeffding-type bonus, which we leave for future work. Thanks for raising this! We will add clarifications and discussions on the referred papers in the revised version.

Reviewer comment (Writing):
Could the authors also provide some motivation examples of CVaR RL under low-rank MDPs? It will help the readers to understand the significance of this setting and how this formulation is related to real-world applications.

Author response:
**Motivation examples**: Thanks for pointing this out. We totally agree that introducing more real-world motivating examples can significantly benefit our presentation. In fact, the family of low-rank MDPs includes models such as linear MDPs [1] and block MDPs [2], which are widely used in practice. For example, block MDPs are considered in [3, 4] to tackle rich and noisy observations in the environment. In addition, the CVaR metric is popular in robust RL [5], distributional RL [6], and portfolio optimization [7]. Due to their abundant applications, we believe studying the sample complexity of CVaR-RL is of unique interest. We will add more discussions in the next version.

[1] Jin, Chi, et al. "Provably efficient reinforcement learning with linear function approximation." Conference on Learning Theory. PMLR, 2020.

[2] Du, Simon, et al. "Provably efficient RL with rich observations via latent state decoding." International Conference on Machine Learning. PMLR, 2019.

[3] Han, Beining, et al. "Learning domain invariant representations in goal-conditioned block mdps." Advances in Neural Information Processing Systems 34 (2021): 764-776.

[4] Efroni, Yonathan, et al. "Provable RL with exogenous distractors via multistep inverse dynamics." arXiv preprint arXiv:2110.08847 (2021).

[5] Hiraoka, Takuya, et al. "Learning robust options by conditional value at risk optimization." Advances in Neural Information Processing Systems 32 (2019).

[6] Achab, Mastane, et al. "Distributional deep Q-learning with CVaR regression." Deep Reinforcement Learning Workshop NeurIPS 2022. 2022.

[7] Krokhmal, Pavlo, Jonas Palmquist, and Stanislav Uryasev. "Portfolio optimization with conditional value-at-risk objective and constraints." Journal of risk 4 (2002): 43-68.

Reviewer comment (Theory):
In Section 4.1, it is mentioned that the algorithm requires two sets of trajectories. Is it true that a *simulator is required* to obtain these two sets? By simulator, I meant the ability to draw samples from the *true* transition probability $P^*_h$. If this is indeed required, it is a major assumption that the authors fail to disclose in the abstract or in a formal statement. I strongly encourage the authors to highlight this fact.

Author response:
**Simulator required**: We want to note that our setting is different from the literature of RL with simulators. Usually, in RL problems, the assumption of the simulator indicates that one can input **any** state-action pair into the simulator, and then the next state would be returned.

Instead, our setting is aligned with the online RL literature, which only requires **interaction** with the environment (i.e., one can roll out a policy in the environment) and doesn’t need the simulator.

Reviewer comment (Theory):
Suppose my understanding of the above is correct. In that case, the result becomes fairly trivial: combining the realizability assumptions with a finite model class (assumption 3.2) with access to a simulator, it is easy to see that given enough samples, any algorithm is no-regret. Nevertheless, there is still credit in proving finite-sample regret bounds in Theorem 4.1.

Author response:
**any algorithm is no-regret**: We note that the reviewer may have misunderstood our setting. As we consider the classical online RL, the learner has to interact with the environment to optimize the policy. We do not have a simulator that can return the next state for arbitrary state-action input pairs. 

It is highly nontrivial to improve the sample complexity in online RL, as shown in the literature (UCB-VI [2], linear MDP [3], bilinear class [4]).  In this case, the notion of no-regret learning refers to attaining a sublinear regret, which can not achieved by an arbitrary algorithm. For example, it is well-known that myopic exploration (e.g., $\epsilon$-greedy) has exponential sample complexity in the worst case [1].

[1] Osband, I., Van Roy, B., Russo, D. J., Wen, Z., et al. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1–62, 2019.

[2] Azar, M. G., Osband, I., & Munos, R. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning (pp. 263-272). PMLR.

[3] Jin, Chi, et al. "Provably efficient reinforcement learning with linear function approximation." Conference on Learning Theory. PMLR, 2020.

[4] Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., & Wang, R. Bilinear classes: A structural framework for provable generalization in RL. ICML 2021

Reviewer comment (Novelty):
The claim that this is the first provably efficient CVaR RL algorithm is incorrect. There have been regret bounds for CVaR RL in function approximation settings in the literature.

Author response:
**The first regret bounds for CVaR RL**: To the best of our knowledge, we present the first sample-efficient algorithm for optimizing the (static) CVaR metric that carefully balances the interplay between risk-averse RL and low-rank MDP structure. Furthermore, we design a computationally efficient planning oracle that makes our algorithm only require polynomial running time with an MLE oracle. 
* We are aware that [7] provides regret guarantees for dynamic CVaR with general function approximation. However, dynamic CVaR is very different from static CVaR. As you have mentioned, these two metrics have non-trivial implications in algorithm design and theoretical analysis.
* We would greatly appreciate it if more related works presenting regret guarantees for static CVaR RL with function approximation could be provided. We are more than happy to soften our contribution arguments. Thanks for raising this.

Reviewer comment (Theory):
The author is encouraged to at least discuss the difference between dynamic and static CVaR, as this has non-trivial implications in algorithm design and theoretical analysis.

Author response:
**Difference between dynamic and static CVaR**: The static CVaR setting we studied is quite different from dynamic CVaR, so comparisons are not covered. Thanks for raising this! We plan to add the following paragraph discussing dynamic CVaR.

* Iterated CVaR, also known as dynamic CVaR, is an important variant of CVaR quantifying the worst $\tau$-percent performance \emph{at each step} of decision-making [5-7]. Such a definition allows the agent to control the risk throughout the decision process tightly. Work [6] studied regret minimization of iterated CVaR in tabular MDPs. A recent attempt [7] studies Iterated CVaR RL with general function approximations and provides regret guarantees using the *eluder dimension* of the function class.

It would be greatly appreciated if you could provide additional valuable references. We are happy to add discussions to our work for a more comprehensive review in the next version.


[5] Chu, S., & Zhang, Y. (2014). Markov decision processes with iterated coherent risk measures. International Journal of Control, 87(11), 2286-2293.

[6] Du, Y., Wang, S., & Huang, L. Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path. ICLR 2022

[7] Chen, Y., Du, Y., Hu, P., Wang, S., Wu, D., & Huang, L. (2023). Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation. arXiv preprint arXiv:2307.02842.

Reviewer comment (Theory):
The proposed and improved algorithms are computationally inefficient in the sense that they still require to call MLE oracles.

Author response:
**Call to MLE**: We thank the reviewer for highlighting the computational aspects of our algorithms involving MLE oracles. In response to this concern, we offer the following clarifications and future directions for refinement:
* In theory, MLE oracles are widely used in model-based analyses (e.g., FLAMBE [4], CPPO [5], REP-UCB [6])
* MLE oracles are also widely used in practice. For example, in RLHF, the reward model is trained based on a maximum likelihood estimator (MLE) [1,2]. They are also widely used in general RL [3]. In practice, the MLE oracle can be reasonably implemented whenever optimizing over $\mathcal{F}$ is feasible (e.g., parameterized by neural networks). Specifically, one can simply use gradient descent to minimize the empirical loss. Therefore, we think MLE oracles are feasible in practice.

[1] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730-27744.

[2] Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.

[3] Farahmand, A. M., Barreto, A., & Nikovski, D. (2017, April). Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics (pp. 1486-1494). PMLR.

[4] Agarwal, A., Kakade, S., Krishnamurthy, A., & Sun, W. (2020). Flambe: Structural complexity and representation learning of low-rank mdps. Advances in neural information processing systems, 33, 20095-20107.

[5] Uehara, M., & Sun, W. Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage. ICLR 2022

[6] Uehara, Masatoshi, Xuezhou Zhang, and Wen Sun. "Representation learning for online and offline rl in low-rank mdps." arXiv preprint arXiv:2110.04652 (2021).

Reviewer comment (Theory):
In Algorithm 1, for the part of collecting two transition tuples, could you explain why using different policies to sample the two actions.

Author response:
**Data collection employed in Algorithm 1**: Thanks for raising this. The use of different policies to sample actions at this stage is intentional, allowing for a more diversified exploration of the model. By doing so, we maintain a balance between exploration and exploitation, a critical aspect in the context of risk-sensitive reinforcement learning. The two distinct datasets, $D_h$ and $D_h'$, have different (marginalized) distributions, serving separate purposes that contribute to our algorithm's proficiency. Specifically, 
 *  The first set, $D_h$, documents the progression of the exploration policy $\pi_{k−1}$ up to step $h$ when rolled out with the initial budget $c_{k−1}$. This procedure helps in designing the bonus terms, which are instrumental in calculating the exploration's effectiveness (Please refer to Lemma C.2 for bonus design rationale).
 * The second set, $D_h'$, is aimed at enhancing the quality of our transition kernel estimates, which is fundamental for improving the overall algorithm's efficiency. By taking two consecutive uniform random actions, we introduce variability that leads to a distinct set of transitions (as further explained in Lemma E.3).