Paper: Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation (ICLR-2024)

Reviewer comment (Novelty):
It seems to me the main contribution of this paper is to transfer and generalize some existing techniques and results (especially in finite horizon setting) to infinite horizon averaged return setting. The definition of AGEC Complexity Measure, FLOP algorithm and the techniques for analyzing lazy update rules seem share much similarity with previous literature like [1], [2]. I didn't find much novelty in algorithm design or technique analysis.

Author response:
- While we admit that the structural assumption over the Bellman dominance shares a similar form as the standard GEC (Zhong et al., 2022), the transferability assumption is new. Moreover, our algorithm design is different from the posterior-sampling-based approach presented in Zhong et al. (2022). We opt for an optimism-based algorithm for establishing a low policy switch updating, which is crucial in the achievement of sublinear regret for AMDPs.  
- Compared with Xiong et al. (2023), we adopt a weaker assumption over the Bellman dominance and provide a more rigorous definition of the eluder condition with nuanced adpation to the average-reward setting. Moreover, the average-reward RL adopts a completely different notion of regret, and the Q- or V- function does not naturally exist due to the infinite horizon without discount. we consider a broad subclass where a modified version of the Bellman optimality equation holds and present a novel regret decomposition for theoretical analysis of the explorability of AMDPs, resulting in significantly different regret analysis from the aforementioned work.

Reviewer comment (Presentation):
There is no conclusion section. I would suggest the authors at least have a few works to summarize the paper and discuss the future works.

Author response:
- Thank you for your valuable suggestions. We have added a conclusion section in the revised version to summarize the paper and discuss future works.

Reviewer comment (Theory):
Is the "lazy policy update" really necessary? Although the authors explain the motivation for low policy switching is because of the additional cost in regret analysis. I'm curious whether it can be avoidable or it reveals some fundamental difficulty.

Author response:
- That's a really interesting question. There are two primary approaches to address online learning of AMDPs: the reduction to episodic MDPs, and the utilization of a low policy switching algorithm. However, achieving $\mathcal{\tilde{O}}(\sqrt{T})$ has proven to be challenging for reduction-based algorithms in existing works under the online learning framework (Wei et al., 2019, Wei et al., 2021), unless fortified with strong assumptions, such as ergodic AMDPs, where every policy is guaranteed to reach every state after a sufficient number of steps. In our pursuit of a near-optimal algorithm without relying on such stringent assumptions, we introduce an additional yet mild structural assumption to ensure the validity of a low-switching algorithm for general function classes. 

- At out best knowledge, all existing algorithms for online learning of AMDPs depend on such lazy update strategy to achieve a near optimal regret beyond strong assumptions, from the classical UCRL2 algorithm for tabular AMDPs (Jaksch et al., 2010), to FOPO for linear AMDPs (Wei et al., 2021) and (UCRL2-VTR for linear mixture MDPs (Wu et al., 2022). Furthermore, we speculate that the underlying mechanism behind this phenomenon lies in the challenge of accurately evaluating $\pi_t$ when imposing a step-by-step policy update, i.e., $\pi_{t}\neq\pi_{t+1}$ for all $t\in[T]$. This difficulty arises due to the intricate dependency structure in the historical data $\{(s_i,a_i)\}_{i\leq[t]}$.

- Specifically, in the episodic setting, the learner executes the policy $\pi_t$ throughout the epside, and the value function can be computed by bellman equation starting from a zero function. Thus, the error can be controlled and is bounded by the cummulative error across episode following decoposition ${\rm Reg}(T)\leq\sum_{t=1}^T\sum_{h=1}^H\mathbb{E}[\mathcal{E}(f_t)(s_h^t,a_h^t)\vert\pi_t]$. This means that the estimation error of the current policy can be controlled by the sum of uncertainty. But for average reward case, if you update policy in each iteration, we cannot bound the suboptimality of the current policy by a sum of estimation errors as ${\rm Reg}(T)\leq\sum_{t=1}^T\mathcal{E}(f_t)(s_t,a_t)+\sum_{t=1}^T\left(\mathbb{E}[V_t(s_{t+1})]-V_t(s_t)\right)$.

- The low-switching policy update in our algorithm ensures that the learner scrutinizes the policy based on data collected from new epochs. This corresponds to the update condition in our algorithm, where we only update when a substantial increase in cumulative discrepancy surpassing $3\beta$ has occurred since the last update.

Reviewer comment (Theory):
In Theorem 3, the definition of $\beta$, is $sp(v^*)$ inside or outside of the log?

Author response:
- The definition of the optimistic parameter should be $\beta=c\cdot{\rm sp}(V^*)\cdot\log(T\mathcal{N}^2_{\mathcal{H}\cup\mathcal{G}}(1/T)/\delta)$, and we have revised the expression in the new version for cleaner presentation (highlighted ). Thank you for pointing out the confusion.

Reviewer comment (Theory):
Why the algorithm is called "Fixed-Point ..."? I'm not very understand why Eq. 4.1 is a fixed-point optimization problem.

Author response:
- If we consider specific problems like linear AMDPs, the constraint of the algorithm can be indeed written into a fixed-point equation with an additional optimistic bonus, which is similar to the FOPO algorithm proposed by Wei, et al. (2021). Nevertheless, we acknowledge the reviewer's concern that this nomenclature might lead to confusion. In response, we have renamed the algorithm as the optimistic Local-fitted Optimization with Optimism (LOOP) in the revised version.

Reviewer comment (Theory):
Big-O notation is used in the definition of AGEC (Definition 3), where the defined quantities $d_G$ and $\kappa_G$ (which also appear in $\mathcal{O}$) are smallest numbers that satisfy the inequalities that involve $\mathcal{O}$. This does not make much sense as a mathematical definition, with $\mathcal{O}$ being asymptotic.

Author response:
- Thank you for pointing out the lack of rigor in our definitions and your comments have been well-taken. In the original version, we used $\mathcal{O}$ notation to hide the contants in the definition, which may not adhere to mathematical rigor. In the revised version, we substitute the $\mathcal{O}$ notation with explicit definitions of absolute constants (highlighted in blue), which does not hurt the correctness of the result.

Reviewer comment (Writing):
The set of discrepancy functions $\{l_f\}_f$ abruptly appears in Definition 3 without any proper definition. In later sections, it is observed that it is an important quantity.

Author response:
- Regarding the dicrepancy function, we would like to highlight that the definition of the discrepancy function is inherently **problem-specific**, making it challenging to offer a unified definition applicable across all cases. To ensure clarity in our presentation, we provide a detailed definition of the problem-specific discrepancy function in each corresponding section such as (3.1), (3.3), (C.2).

Reviewer comment (Theory):
The function approximation error can be critical in RL, as its multiplying factor usually depends on the exploration performance in various forms. If the realizability assumption is removed, how does the additional term depend on the complexity measure defined in this paper?

Author response:
- We would like to clarify that our complexity measure characterizes the difficulty in exploring the best-in-class model within the designated hypothesis class. On the other hand, the approximation error stems from the disparity between the best-in-class approximator and the ground-truth optimal function, which is not theoretically provable. Consequently, the removal of the realizability assumption will sure introduce additional error terms; however,  **it does not depend on the proposed complexity measure**.  
- A generalization of realizability for value-based problems is that there exists $f\in\mathcal{F}$ and $\epsilon_{\rm real}>0$ such that $\Vert Q^*-Q_f\Vert_\infty\leq\epsilon_{\rm real}$. The condition requires that the optimal bias state-action value function lies in the function class approximately with $\epsilon_{\rm real}$ error, enabling a more nuanced analysis on approximation. In this case, we conjecture that our algorithm, designed with a larger $\beta$ to accommodate misspecification, is capable of finding an $(\epsilon + \mathrm{poly}(d, \mathrm{sp}(V^*)) \cdot \epsilon_{\rm real})$-optimal policy using polynomial (in $\epsilon$) samples. It is indeed a very interesting future direction to design a unified sample-efficient algorithm considering the approximation error under general function approximation. We leave more meticulous analysis as future work.

Reviewer comment (Writing):
In Equation 2.1, what is $V^*(s,a)$? Should it be $Q^*$?

Author response:
- Thank you for pointing that out. We have rectified the typo (Page 3), which should be 
$$J^*+Q^*(s,a)=r(s,a)+\mathbb{E}_{s'\sim\mathbb{P}(\cdot|s,a)}[V^*(s')].$$

Reviewer comment (Theory):
In the abstract and in multiple places in the paper, $sp(v^*)$ appears with $v^*$. Should it be $sp(V^*)$? In the paper, it is assumed that $sp(V^*)$ is known. Is this knowledge necessary?

Author response:
- Thank you for pointing out the confusion in the notations. We have now replaced ${\rm sp}(v^*)$ by ${\rm sp}(V^*)$ for a clearer presentation in the revised version. Regarding the span ${\rm sp}(V^*)$, we assert that the knowledge of the span of the optimal state bias function is crucial for **establishing a more rigorous regret guarantee** due to theoretical necessity. It is noteworthy that, unlike the episodic setting, there is no inherent upper bound for the optimal function $V^*$ in the average-reward setting, leading to a hypothesis class with uncontrollable complexity. The assumption of a known span implies a bounded span for the optimal bias function, with ${\rm sp}(V^)$ being available to the learner. This knowledge allows for the construction of a more stringent optimistic parameter. Alternatively, in practical scenarios, the learner can choose a **conservative estimation** of the span, such as a sufficiently large constant $B_{sp}>0$, ensuring $B_{sp}\geq{\rm sp}(V^*)$. The substitution of ${\rm sp}(V^*)$ with $B_{sp}>0$ in the construction of $\beta$ ensures the persistence of the optimistic planning in Algorithm 1, and results in an inflation of regret in squared root.

Reviewer comment (Evaluation):
Can authors provide a brief comparison between this work and existing works in terms of the regret results?

Author response:
We would like to thank the reviewer for the positive assessment of this work! Here's our response to the question.

> Can authors provide a brief comparison between this work and existing works in terms of the regret results?

- Please see our general response for details.

Reviewer comment (Novelty):
The sample complexity is based on GEC and is not entirely novel. Specifically, Definition 3 (Bellman Dominance) in this paper is of the same form with Definition 3.4 of Zhong et al. (2022b), where $d$ is defined such that the sum of the Bellman error being less than the in-sample training error plus the burn-in cost.

Author response:
-  While we admit that the structural assumption over the Bellman dominance shares a similar form as the standard GEC (Zhong et al., 2022b) with nuanced adaption in average-reward terms, the transferability assumption is a novel addition. This additional structural constraint presents the unique challenges of AMDPs, and plays a pivotal role in facilitating the design of a low-switching algorithm.  This is particularly essential in addressing the inherent difficulty in exploring AMDPs. 
-  Moreover, our algorithm design diverges from the posterior-sampling-based approach presented in Zhong et al. (2022b). We embrace an optimism-based algorithm to establish a low policy switch updating mechanism, a crucial element in achieving sublinear regret for AMDPs. As a consequence, the technical details of our regret analysis differ significantly from the aforementioned work. Please refer to our general response more desciptions of technical novelties.

Reviewer comment (Theory):
The authors introduce the discrepancy function in the definition of AGEC, and shows a simple example of discrepancy function being the Bellman error for the value-based case, and $(r_g+P_g V_{f'})(s_t, a_t)-r(s_t, a_t)+V_{f'}(s_{t+1})$ for the model-based case. However, there seems a lack of discussion regarding alternative choices for the discrepancy function, such as the Hellinger distance-based discrepancy in Zhong et al. (2022b).

Author response:
- We appreciate the reviewer's suggestion regarding the importance of discussing alternative choices for the discrepancy function, including the Hellinger distance-based discrepancy. Discussions about MLE-AGEC concerning an alternative discrepancy functions are presented in **Appendix C** (highlighted in blue) with a different transferability assumption over the structure. Here, we choose the total variation (TV)-based discrepancy function instead, which results from the slight difference between the choice of discrepancy: $l_{f'}(f,g,\zeta_t)=\frac{1}{2}|P_f(s_{t+1}|s_t,a_t)/P_{f^*}(s_{t+1}|s_t,a_t)-1|$ (ours) and $l_{f'}(f,\zeta_t)=\frac{1}{2}(P_f(s_{t+1}|s_t,a_t)/P_{f^*}(s_{t+1}|s_t,a_t)-1)^2$ (Zhong et al., 2022b). We remark that both choices of discrepancy is sufficient to cover MLE-based algorithms.

Reviewer comment (Novelty):
The algorithm is based on upper confidence bound, which the confidence region chosen based on the discrepancy function. This approach is closely related to Algorithm 1 in Chen et al. (2022b) and Algorithm 1 in Jin et al. (2021).

Author response:
- While acknowledging the resemblance of our algorithm design to previous lines of work on optimism-based algorithms for general function approximation, we notably diverge by employing a low-switch updating strategy coupled with a discrepancy-based updating condition.
- Moreover, we would like to comment that our algorithm is in some sense information theoretical -- our work identifies a statistically tractable subclass of AMDPs, and the algorithm **serves as a certificate for the regret upper bound**. The main focus of our paper is to address the problem of minimal structural assumptions that empower sample-efficient learning in AMDPs, which facilitates algorithm design beyond specific problems such as linear AMDPs and linear mixture AMDPs.

Reviewer comment (Theory):
The proposed algorithm is usually impractical to implement, since it involves solving a global constrained optimization.

Author response:
- While acknowledging the resemblance of our algorithm design to previous lines of work on optimism-based algorithms for general function approximation, we notably diverge by employing a low-switch updating strategy coupled with a discrepancy-based updating condition.
- Moreover, we would like to comment that our algorithm is in some sense information theoretical -- our work identifies a statistically tractable subclass of AMDPs, and the algorithm **serves as a certificate for the regret upper bound**. The main focus of our paper is to address the problem of minimal structural assumptions that empower sample-efficient learning in AMDPs, which facilitates algorithm design beyond specific problems such as linear AMDPs and linear mixture AMDPs.