Paper: Horizon-free Reinforcement Learning in Adversarial Linear Mixture MDPs (ICLR-2024)

Reviewer comment (Theory):
The occupancy measure-based algorithm is not computationally efficient as the running time has polynomial dependence on the state number $S$ and action number $A$.

Author response:
We agree with the reviewer as we have discussed in Remark E.2. The necessity of computing all the $\Omega(S)$ entries of occupancy measure is the limitation of our algorithm. However, as we have shown in Section 6, using occupancy measure is the key technique to get rid of the $H$ dependency in online mirror descent regret. We will explore more efficient algorithms in the future work.

Reviewer comment (Theory):
I think it should be clearer that also homogenous transition dynamics are required for obtaining reward free bounds. Therefore, the Bellman optimality equation at page 3 should not have $h$ in the footnote of the operator $\mathbb{P}$.

Author response:
**A1**: Thank you for your suggestion and we have fixed it in the reversion.

Reviewer comment (Writing):
The value function $\overline{V_{k,1}}$ is never formally defined in the paper. So it is difficult to understand what it denotes when reading the regret decomposition in equation (6.1). If I understood correctly from the Appendix, each mirror descent iterate $z_k$ induces via the marginals a transition kernel $\overline{p_{k}}$ and a policy $\pi_k$. At this point $\overline{V_{k,1}}$ denotes the initial value of policy $\pi_k$ in the MDP endowed with reward function $r_k$ and transition dynamics $\bar{p}_k$. Can the authors confirm that this is correct and if yes add it to their revision?

Author response:
**A2**: We are sorry for the missing formal definition. We have added an equation to formally define $\bar{V}_k,1$ in the revision.

Reviewer comment (Writing):
The definition of Regret at page 3 is a bit unclear. Indeed saying that $V^\star_k$ is the optimal state value function could make the reader believe that $V^\star_k = \max_{\pi} V^{\pi}_k$, that is the regret we control has the maximum inside the sum. However, the regret controlled in the paper has a fixed comparator policy which might not be optimal for any of the reward function revealed at each round.

Author response:
**A3**: Thank you for your suggestion and we have revised it to make it clearer.

Reviewer comment (Theory):
I think that it is unclear that $I^k_h$ defined in Appendix C.2 is decreasing. After inspecting the proofs I think that what the authors need is that for any fixed $k$ then $I^k_h$ is decreasing with respect to $h$. Is this correct?

Author response:
**A4**: Yes, we indeed need for any fixed k then $I_k^h$ is decreasing with respect to $h$. We apologize for the confusion and have made it clear in the revision.

Reviewer comment (Theory):
Is it possible to design policy optimization algorithms for this problem setting?

Author response:
**A1**: We would like to clarify that policy optimization has previously been applied to adversarial linear mixture MDP [1][2]. However, none of these approaches can guarantee a horizon-free regret upper bound. The reason for this limitation has been shown in Section 6. Specifically, the regret for online mirror descent is $\tilde{O}(H\bar{L}\sqrt{K})$, where $\bar{L}$ is the upper bound of the subgradient. When employing (multiplicative weights update/EXP3 type) policy optimization, due to reward accumulating, the scale of the Q-functions and the correspoding subgradient is $\bar{L}=O(1)$, which will inevitably introduce a dependency on $H$. Therefore, we resort to occupancy measures to tackle this challenge.

Reviewer comment (Theory):
Is it possible to avoid the usage of occupancy measure (which is not quite efficient in real world).

Author response:
**A2**: To our knowledge, to tackle adversarial reward, existing approaches are based on either policy optimization or occupancy-measure. As we found that policy optimization is difficult to deduce a horizon-free guarantee, we show that a feasible way is to take occupancy measure-based approach. We will explore other approaches rather than occupancy measure in the future work.