Paper: Optimal Sample Complexity for Average Reward Markov Decision Processes (ICLR-2024)

Reviewer comment (Novelty):
The result is somewhat thin in the sense that it feels like filling a small gap that was somehow overlooked by several previous groups of researchers, though I personally like the cleanness of the result.

Author response:
Contribution: The optimal sample complexity for AMDP has been recognized as a significant and challenging open problem in RL theory, motivating many attempts and improvements over the last 6 years (see Table 1). Our algorithm is the first one in the literature that achieves minmax optimal sample complexity for uniformly ergodic MDP. This is achieved by bringing in multiple algorithmic ideas and analysis techniques. We highlight this contribution in our revised abstract and in the third paragraph of the introduction.

Reviewer comment (Theory):
I'm a little confused about some notation. Is alpha in (2.6) in R^|S|? It should be, but later in the definition of \bar{alpha}, we need to choose the maximum alpha, which seems incorrect since it is a vector.

Author response:
Definition of $\alpha^{\pi}$: The $\alpha^{\pi}$ is the long run average reward. In general, it is indeed a function $S\rightarrow R$. However, because we assume that all policies induce mixing (Assumption 1), the long-run average reward induced by any policy is independent of the initial state. Therefore, as explained between equations (2.3) and (2.4), we define it as a limit $\alpha^\pi\in R$.

Reviewer comment (Writing):
Equation (2.4), where is \eta defined? It seems to appear without any context.

Author response:
$\eta(\cdot)$ definition: Thanks for pointing this out! We made changes to the manuscript, and now $\eta_\pi$ is the stationary distribution of $P_\pi$.

Reviewer comment (Experiments):
Can you comment on the choice of distribution of Z(s,a)? Does it make a difference if another distribution (non-uniform) is used?

Author response:
Uniform distribution of $Z$: $Z$ need not be uniformly distributed. It only serves as a perturbation so that the optimal value is unique with a positive optimality gap. This technique is introduced by Li et al. (2022). We added a short discussion about this after equation (3.1).

Reviewer comment (Experiments):
Though I can understand the result is theoretical, have the authors used any numerical results to verify the optimal algorithm behavior that suggests the sample complexity scaling?

Author response:
Numerical experiments: Thanks for the suggestion. We added two experiments that separately verify our algorithm's complexity dependence on $\epsilon$ and $t_{mix}$, yielding results that are consistent with our theory's prediction.

Reviewer comment (Theory):
What are the main barriers to providing an algorithm with dependence not on a mixing time-type quantity but on the span of the optimal value? This question has its importance because it is known that this guarantee will be strictly tighter than mixing dependent.

Author response:
As you suggest, the parameter $H$ has advantages over $t_{mix}$ as the basis for an upper bound. We included a short discussion of this in Section 5. Moreover, we have evidence suggesting that the natural analog of our algorithm should have a sample complexity upper bound of $\widetilde O (|S||A|H\epsilon^{-2})$, if we replace the uniform ergodicity assumption by one that assumes the span semi-norm of the transient value function induced by any policy is bounded by $H$. However, we didn't adopt this setup for the following reasons. First, $t_{mix}$ places a direct assumption on the primitives---namely the transition kernel---of the AMDP, yet the assumption on $H$ involves the solution to Poisson equations. Second, we are aiming at a "worst case" complexity result for the AMDP. The $H$ metric, considering the interplay between the reward and the transition kernel, has the flavor of instance-dependent complexity theory (see Khamaru et al. (2021), cited in the paper).

Reviewer comment (Theory):
Is it possible to provide an algorithm without a reduction to the discounted setting?

Author response:
Thanks for asking about this. Before giving a direct response, we would like to point out that the idea of solving an AMDP by reducing it to a DMDP and applying dynamic programming has been considered and implemented since the 1970s, see Hordijk and Tijms (1975). In fact, the optimality of our DMDP algorithm implies that one can choose $\gamma$ of the approximating DMDP to be much smaller than that in Jin and Sidford (2021), where $(1-\gamma) = \Theta(\epsilon/t_{mix})$, and still get the optimal sample complexity. This is not the case in Jin and Sidford (2021), where enlarging $\gamma$ will worsen the sample complexity. However, note that choosing large $\gamma$ is inefficient from a computational standpoint, because solving the empirical MDP with large $\gamma$ is typically hard (consider using value iteration), and hence is not implemented in the paper. Therefore, the reduction method is mainly a computational tool. So, statistically speaking, we think it is possible to achieve the same sample complexity using a model-based approach without the reduction. However, this might lead to computational complications, e.g. slow convergence rate for non-mixing empirical AMDPs.

Reviewer comment (Theory):
Is it possible to extend this approach to the exploration setup and provide a feasible algorithm with guarantees like (Ortner, 2020)?

Author response:
The key ideas and bounds extend to the online setting without any issue. As our answer to your question (2) suggests, our bound is not reliant on a particular choice of the discount $\gamma$. Therefore, we do believe that our theory can be used to extend classical model-based online discounted RL algorithms to the average reward setting.

Reviewer comment (Novelty):
The algorithm is primarily a synthesis of methodologies from Jin & Sidford (2021) and Li et al. (2020). While this approach has its merits, the novelty of this paper is somewhat limited given its dependence on previous works.

Author response:
Thanks for your comments and suggestions! We address your concerns and revise the paper accordingly as follows: 
1. Contribution: The optimal sample complexity for AMDP is a significant and challenging open problem in RL theory. A number of related publications in notable venues have appeared in the last six years (see Table 1). We offer a new algorithm and a novel analysis that combined yield the provably first min-max optimal sample complexity method for uniformly ergodic MDPs. We strongly believe that our result holds significant theoretical value and is therefore worth publishing. In fact, it is hard to argue the opposite in view of the importance of this problem. We acknowledge the useful ideas introduced in earlier papers. But clearly, key (non-trivial!) additional insights were required to solve this problem; otherwise, a minmax optimal result would have been offered already by one of the previous authors who have worked on this problem. Our contribution requires skillfully combining and extending multiple algorithm ideas and analysis techniques. We highlight these contributions earlier in the paper, in our revised abstract, and in the third paragraph of the introduction.

Reviewer comment (Novelty):
The paper could be enhanced by placing greater emphasis on the challenges addressed by the study and the innovative aspects of the proposed algorithm. Highlighting these elements would help to showcase the unique contributions of the paper and further establish its significance in the field.

Author response:
Thanks for your comments and suggestions! We address your concerns and revise the paper accordingly as follows: 
1. Contribution: The optimal sample complexity for AMDP is a significant and challenging open problem in RL theory. A number of related publications in notable venues have appeared in the last six years (see Table 1). We offer a new algorithm and a novel analysis that combined yield the provably first min-max optimal sample complexity method for uniformly ergodic MDPs. We strongly believe that our result holds significant theoretical value and is therefore worth publishing. In fact, it is hard to argue the opposite in view of the importance of this problem. We acknowledge the useful ideas introduced in earlier papers. But clearly, key (non-trivial!) additional insights were required to solve this problem; otherwise, a minmax optimal result would have been offered already by one of the previous authors who have worked on this problem. Our contribution requires skillfully combining and extending multiple algorithm ideas and analysis techniques. We highlight these contributions earlier in the paper, in our revised abstract, and in the third paragraph of the introduction.
2. Main challenge: A crucial idea that allows us to achieve optimality, compared to the previous approaches, is the realization that the optimal complexity dependence for Discounted MDPs (DMDPs) should be $\widetilde O(t_{mix}(1-\gamma)^{-2}\epsilon^{-2})$, c.f. Wang et al. (2023). However, from an algorithm design standpoint, this is not enough. This needs to be coupled with a sufficiently small initialization sample size. Therefore, the main challenge to our algorithm design and analysis is to achieve a $\widetilde O(t_{mix}(1-\gamma)^{-2}\epsilon^{-2})$ sample complexity with a small minimum sample size. This is explained in the Introduction, Sections 3, and 3.1. These are key insights that are non-trivial to obtain and exploit in algorithmic design, as we explain and illustrate in the numerical results added in the paper.

Reviewer comment (Writing):
The paper could benefit from a greater emphasis on the challenges addressed by the study. Could you provide more information on the specific challenges inherent to the problem you are solving and how your approach effectively addresses these issues?

Author response:
Thanks for your comments and suggestions! We address your concerns and revise the paper accordingly as follows: 
2. Main challenge: A crucial idea that allows us to achieve optimality, compared to the previous approaches, is the realization that the optimal complexity dependence for Discounted MDPs (DMDPs) should be $\widetilde O(t_{mix}(1-\gamma)^{-2}\epsilon^{-2})$, c.f. Wang et al. (2023). However, from an algorithm design standpoint, this is not enough. This needs to be coupled with a sufficiently small initialization sample size. Therefore, the main challenge to our algorithm design and analysis is to achieve a $\widetilde O(t_{mix}(1-\gamma)^{-2}\epsilon^{-2})$ sample complexity with a small minimum sample size. This is explained in the Introduction, Sections 3, and 3.1. These are key insights that are non-trivial to obtain and exploit in algorithmic design, as we explain and illustrate in the numerical results added in the paper.