Paper: Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR-2024)

Reviewer comment (Writing):
I still lack some intuition for how the policy class is constructed, and why it is hard to learn.

Author response:
We thank reviewer ycxM for their valuable comments and feedback.

$~$
## C.1 Diagrams and Intuition for Policy Class

We have uploaded a new pdf version of the paper that now contains a section with diagrams illustrating the construction of the hard MDP instance (Appendix D.5, pages 28-29) as well as a diagram of hyper-spherical sectors in Appendix B. We hope this provides some clarity and better intuition.

Reviewer comment (Writing):
Minor typo: on page 21, the definition of $W$ needs to have \exp. Also, where is $g(\gamma)$ defined? I see it used in Lemma C.5, but I wasn't sure if it was properly defined anywhere.

Author response:
$~$
## C.3 Typos

Thank you for pointing this typo out! As for $g(\gamma)$, it is defined in the statement of Lemma C.5 but in fact we use it before this (e.g.\ statement of Lemma C.4) - **we will properly define it earlier on in Appendix C**.

Reviewer comment (Theory):
It wasn't obvious to me why finite horizon linear MDPs can be solved with number of deployments independent of $d$. More generally, is the gap really between finite horizon vs infinite horizon, or between linear MDPs and your $Q^\pi$ realizability assumptions (4.1 and 4.2)?

Author response:
$~$
## C.4 Finite Horizon

**Finite-horizon linear MDPs can even be solved in the offline setting** ($K=1$) with a total of $d \cdot H$ queries (where $Q^\pi_t$ realizability is assumed at each step $t \in \{1, ..., H\}$). This can be seen by at each step $t$ selecting $d$ queries whose features form a basis and then running a backward induction (See Appendix D.2 of Zanette (2021) for details). 

Therefore, under our $Q^\pi$ realizability assumption, **there is a gap between infinite horizon and finite horizon MDPs**. There may also exist a gap between linear MDPs and our $Q^\pi$ realizability assumption, since the former is a stronger assumption.

Reviewer comment (Theory):
Do you expect to be able to improve the lower bound significantly? Perhaps, to log d or even some polynomial in d?

Author response:
$~$ 
## C.5 Improving the Lower-Bound

See C.2 for a discussion of what is required to improve the lower-bound.

As we have stressed in the conclusion section, it is unclear if the $\log \log d$ dependence on $d$ is tight, which we leave as an open question. We add here that the subspace packing result we use currently is not tailored to our setting in the sense that we associate a learner's query with an entire subspace and use that all vectors in that subspace are far enough in inner product from vectors of any other subspace in the packing. But we only need this for the query not the entire subspace associated to the query. Therefore there is room for improvement with some more specific subspace packing reasoning but it remains unclear if this would allow an improvement to $\log d$ or higher.

Reviewer comment (Theory):
The importance of the proposed problem setting is unclear. The problem setting suggests that if the number of adaptivity rounds is too small (less than $\log\log d$), the $d$-dimensional feature space cannot be fully spanned by the given features induced by the data obtained so far, and the solution $\theta$ that admits the value function (either the optimal value function or the value function associated with the target policy to be evaluated) is not unique, leading to a large value gap when considering the worst state-action pair to be evaluated. However, it seems that such a separation gap becomes unimportant if a regret guarantee is admitted instead of a worst-case evaluation error guarantee (as defined in Def 3.1). For the regret guarantee, since only the value gap over existing state-action pairs is relevant (due to the definition of regret) rather than the maximum-achieving state-action pair that may never be encountered (again, due to Def 3.1), the nonuniqueness of $\theta$ will not affect the regret as it affects the policy evaluation error. The question posed is whether such a lower bound does not hold if the definition in Def 3.1 is slightly revised from $P(\sup_{a \in \mathcal{A}}(\cdots))$ to $P(\mathbb{E}_{a \sim \mu}(\cdots))$, where $\mu$ is some distribution over $\mathcal{A}$.

Author response:
The mentioned weakness on considering a worst-case evaluation error (instead of a regret guarantee) (point 1.) **does not apply to the best policy identification** (BPI) problem for which there is no worst-case evaluation (Definition 3.2). For the policy evaluation (PE) problem, it is true that the lower bound no longer holds for the proposed definition of Definition 3.1. Though this is not the setting we study and our definition (and results) remains relevant **if the interest is a worst-case policy evaluation error**.

Reviewer comment (Novelty):
The separation gap may become unimportant if there is any limitation on the value gap (e.g., the sub-optimality gap defined in [1]), or on the size of the action space $|\mathcal{A}|$, or if some coverage assumption holds that assumes the obtained data can cover the target policy. If any of these assumptions hold, the separation gap does not hold either. Since these assumptions are considered mild and hold in many settings, the proposed result in this paper becomes less interesting.

Author response:
As for point 2. on the additional assumptions not covered by our results, it is also **common to have an infinite action space and no additional assumption on the sub-optimality gaps**. Additionally, while it may be necessary to have coverage assumptions in an offline setting where the learner has no control over the data collection, this **assumption need not be made when data is collected adaptively**, as in our setting, since the learner is selecting the state-action pairs to query. For PE, in our second result (Theorem 4.5), the target policy can be fully revealed before collecting any data so the learner can choose the data with knowledge of the target policy (and indeed the hardness is introduced by the worst-case evaluation error). It is an **interesting direction to understand if our results continue to hold under these additional assumptions** or if these assumptions lead to a difference in the sample-efficiency of low-adaptivity RL. We leave this as future-work and will include a discussion about these assumptions in the final version.

Reviewer comment (Novelty):
The problem setting and hard instance construction are very similar to that in Zanette 2021, which diminishes the technical contribution of this work. More comparisons between the technical challenges faced in this work and those in Zanette 2021 would be welcomed.

Author response:
Please refer to Section A.6 in response to reviewer 7s3L for an additional comparison that we will include in Section 5. 

The technical novelty of our work is using (as mentioned by reviewer 7s3L)  **``novel tools such as subspace packing with chordal distance" (Soleymani \& Mahdavifar, 2021) and applying them to our setting** (see Appendix C). These tools have not been applied in an RL setting before as far as we are aware. In particular, the technical challenges include the following points.
* 1. Relating the subspace packing result with chordal distance to the existence of a subspace $B_k$ that is far (does not intersect the $\gamma$-hyperspherical cones) from all the learner's queries up to round $k$ if there are polynomial in $d$ number of queries (Section C). Information can then be erased in the directions of the subspace $B_k$ whose dimension is large if $k \neq \Omega(\log \log d)$. 

The work of Zanette (2021) only requires showing the existence of a $1$-dimensional subspace. A **volumetric argument** can be used in their setting: since the sum of the volumes of an exponential (in $d$) number of hyper-spherical cones is less than the volume of the unit-sphere, there must exist a point in the sphere not covered by these exponentially many cones. This point can be used as the 1-dimensional subspace. However, **we cannot proceed with a similar volumetric argument** because we consider $m$-dimensional subspaces. The space in the sphere left vacant by the cones of the learner's queries can easily be used to argue the existence of individual points not covered by the cones, but not the existence of subspaces of multiple dimensions. Instead, as mentioned above, we adapt tools from the theory of subspace packing.
    
* 2. Using the existence of these subspaces to extend the hard instance construction of Zanette (2021) to our setting over multiple adaptive rounds, while preserving the realizability assumptions. The extension over multiple rounds requires a careful treatment of the dependence of the queries on observed feedback (see Appendix D.4.2/E.4.2) to establish the existence of these subspaces, which does not arise when considering a single round as in Zanette (2021). 

We have uploaded a new pdf version of the paper that now contains a section with diagrams illustrating the construction of the hard MDP instance (Appendix D.5, pages 28-29). We hope this provides some more clarity on the technical differences with Zanette (2021).

Aside from the technical novelties outlined above, we would like to stress the novelty of our main result, which is to show the need for dimension-dependent adaptivity in RL under linear function approximation.

Reviewer comment (Theory):
As mentioned before, the paper provides a lower bound for K, but it is not clear whether it is tight. loglogd is also sometimes considered as a constant in some works, since lnln 10^9 is only approximately 3.

Author response:
The significance of the presented result is in the **existence of a dependence on the dimension $d$** for the amount of adaptivity $K$. From a theoretical perspective, this represents an important difference to $K$ being constant with respect to $d$. Our results provide an **important step in studying the limits of RL under low-adaptivity by showing a dependence on dimension is unavoidable** but much is left to understand. In particular, as we stress in the conclusion, it remains unclear if the $\log \log d$ dependence on $d$ is tight, which we leave as an open question.

Reviewer comment (Theory):
Section 4: “we assume gamma\geq sqrt{3/4}”. Can the authors provide some justification/context for this assumption? Is it purely an assumption to make the analysis work?

Author response:
It is an assumption to **make the analysis work**. In the analysis, we show the existence of subspaces that are at least some chordal distance away from each other and the distance depends on $g(\gamma) = 2\gamma^2 - 1$. We assume $\gamma \geq \sqrt{3/4}$ to ensure $g(\gamma)$ is not too small, which allows us to get the right dependence on the dimension $d$. Given that the discount factor $\gamma$ in discounted MDPs is considered to be close to 1, this **assumption is not restrictive**.

Reviewer comment (Writing):
The reviewer suggests that the authors avoid copying and pasting proofs so that the proof can be shorter.

Author response:
The proofs of both Theorems are based on the same idea of the existence of a sequence of subspaces but the way the constructions of the MDP classes incorporate this sequence of subspaces is different. Therefore once the proofs have been reduced to using the existence of this sequence of subspaces, they are essentially the same. In the final version, **we will better unify these sections** to avoid repetition.

Reviewer comment (Writing):
The reviewer does not quite understand the second paragraph of Section 4, partly because it is a short paragraph with lots of information. In particular, the authors mentioned they established a *matching* lower bound for some problem while the upper bound is d. How does this lower bound relate to the main result?

Author response:
"I don’t quite understand the second paragraph of Section 4" - we believe you meant Section 4.3 but please correct us if not.

This paragraph is related to the **fully-adaptive** setting where adaptivity constraints are removed and the number of batches $K$ is equal to the number of queried data-points $n$. This is in contrast to our main results (Theorems 4.4 and 4.5), which show $K > \log \log d$ are needed to solve BPI/PE when $n \gg K$ but $n$ is polynomial in $d$. In Appendix A, we show that in the fully-adaptive setting, $n \geq d$ is needed to solve BPI/PE to arbitrary accuracy (and therefore $K\geq d$ since $K = n$). We also show a matching upper bound of $K=n\leq d$ to fully solve a PE problem under an assumption on the action space, hence the matching upper and lower bound.

Therefore the **relation between the lower-bound in the fully-adaptive setting and the main results is that they consider different settings of how $K$ relates to $n$**.

Reviewer comment (Novelty):
At the beginning of Section 5, the authors mentioned that their proof extends that in Zanette (2021). However, this paper is not mentioned at all in the rest of this section. It would be great if the authors can compare the idea here with Zanette (2021) in the proof sketch section.

Author response:
In the final version, with the additional allowed page, we will include this comparison to the work of Zanette (2021) in Section 5:

**Comparison to Zanette (2021):** The work of Zanette (2021) erases information along a 1-dimensional subspace ($X$ is of rank $d-1$). Therefore after having observed one round of feedback, the learner only needs a single query to fully distinguish the MDP and solve the problem. Our constructions erase information along $m$-dimensional subspaces where we attempt to maximise $m$ so that even after having observed feedback revealing this subspace, a polynomial number of queries remains insufficient to distinguish the MDP and solve the problem, more adaptive rounds are needed.