Paper: On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR-2024)

Reviewer comment (Theory):
One weakness is that the double descent behaviour is said to be related to the behaviour of the correction factor $\delta$ but I do not think that this is demonstrated analytically. What is precisely happening to $\delta$ when $N=m$, or alternatively what is happening to the difference between the empirical MSBE and the true MSBE? This is one of the main selling points of the paper so it would be good to expand on this.

Author response:
A precise analytical characterization of the relationship between the correction factor and double descent is challenging because of the complex nonlinear dependencies. To the best of our knowledge, it is not yet available even in the case of supervised learning (which is covered by our Theorems as the special case $\gamma=0$). We do show analytically in Sect. 5 and Appendix C that the correction factors $\delta, \Delta, \hat \Delta$ vanish as $N/m \to \infty$ (over-parameterization) or as the $l_2$ regularization parameter $\lambda \to \infty$. 

To shed more light on the difference between the asymptotic empirical MSBE and the asymptotic true MSBE, we added a reformulation of our theorem in Appendix C that highlights the similarities between the second-order correction terms $\hat \Delta$ and $\Delta$.

Reviewer comment (Theory):
"We use the regularized LSTD algorithm, [...] because it converges to the same solution as other TD algorithms." This is true without regularization, but is it true with regularization?

Author response:
Yes, it is also the case! We have added the citation [1].

Reviewer comment (Writing):
Section 3 "Notations" paragraph: what is the "Euclidean norm" of a matrix? Do the authors mean the spectral norm or the Frobenius norm?

Author response:
Yes, it was indeed the operator norm.

Reviewer comment (Presentation):
Items 4 and 5 in the "Contributions" paragraph both repeat that the correction terms vanish when the regularization increases and/or the number of unvisited states go to zero.

Author response:
We reformulated the claim to clarify the different aspects.

Reviewer comment (Writing):
The notation for feature matrix $\Sigma_S$ is unfortunate since it clashes with typical notation for covariance matrices. Is there a reason not to use the more conventional $\Phi$ notation for this matrix?

Author response:
We adopted the same notation as [2] for the matrix $\Phi$ and $\Sigma$.

Reviewer comment (Writing):
Section 3 "Linear Temporal-Difference Methods" section: the Mean-Square Projected Bellman Error is undefined.

Author response:
We have included a citation to guide readers to a more comprehensive explanation of the differences between the Mean-Square Error and the Mean-Square Projected Bellman error [3].

Reviewer comment (Theory):
Section 4.1: For the sampling paradigm I do not think the authors literally mean $(s,s') \sim \mu(s) P(s,s')$. This latter expression would be the marginal probability of observing $s'$ only, and is just a number (not a distribution one can sample from).

Author response:
Thank you for pointing out this problem. We have updated the notation.

Reviewer comment (Experiments):
Section 4.1: "We consider the on-policy setting, where $D_{train}$ is derived from a sample path of the MRP or its stationary distribution $\pi$. These are very different (one is i.i.d.); are both truly considered?

Author response:
We just need one of this sampling to satisfy Assumption 3. Furthermore, it is easier in the on-policy setting to get Assumption 2.

Reviewer comment (Writing):
Some vector norms are indexed by $\pi$, but many other norms are not indexed (e.g. Equations 8, 9, 10). Is this a typo, or are all unindexed norms the Euclidean norm?

Author response:
All unindexed vector norms refer to the Euclidean norm; we have extended the 'Notations' section to clarify this point. We have also added further clarification to the definition of the empirical MSBE.

Reviewer comment (Theory):
Relatedly: the closed form for the LSTD estimator is defined without the usual diagonal $D_\pi$, which implies that the algorithm performs regression on the Euclidean norm rather than $L_2(\pi)$ (one can compare with the more classical weighted version e.g., in Amortila, Jiang, Szepesvári ICML 2023). In this case the LSTD estimator is different from the usual weighted one. Can the authors discuss whether this is a meaningful difference and/or whether the results would hold for the weighted estimator?

Author response:
For regularization, we adopted the definition of regularized LSTD used by [3, 4, 5, 6] where the regularization parameter is applied to the empirical MSBE. Considering a different regularization parameter could alter the solution produced by the algorithm and, consequently, the result of the theorems.

Reviewer comment (Theory):
"Under the assumption that all states are visited $\hat{MSBE}(\theta)$ converges to $MSBE(\theta)$ with probability 1, as the number of collected transitions $n \rightarrow \infty$ (Bradtke & Barto, 1996)." I do not think that this is shown in that paper, which instead deals with the projected Bellman error.

Author response:
Thank you for pointing this out; we updated the sentence with the correct citation.

Reviewer comment (Writing):
Typo in Assumption 3: should be $O(1/\sqrt{m})$.

Author response:
Thank you for pointing out this typo.

Reviewer comment (Theory):
Why are the two asymptotic quantities the parameter number and the number of unvisited states as opposed to the parameter number and just the sample size?

Author response:
1 - In the case of continuous systems, both the sample size $n$ and the number of visited states $m$ are usually equivalent. However, this is not true for discrete systems, where the same states can be sampled multiple times. 
Using $m$, we cover both discrete and continuous cases.

Reviewer comment (Theory):
What are the technical difficulties when studying regularized LSTD for policy evaluation as opposed to regularized least squares for the regression case ($\gamma = 0$)? The text mentions that there is a lack of positive definiteness in the LSTD matrix — is this the only difficulty? How is this difficulty overcome?

Author response:
2 - The main challenge in extending the results from Supervised Learning to RL stems from the absence of positive definiteness in $Q_m(\lambda)$. Without this property, there is no guarantee that the eigenvalues are positive reals, which would yield the existence of the resolvent and the fixed-point $\delta$, bound the operator norm of the resolvent, address some technical aspects in the proofs, and help interpret the correction terms. Fortunately, even though $Q_m(\lambda)$ is not positive-definite, its unique structure still allows us to derive these results, albeit in a more complex manner. For instance, we exploited that the matrix  $H(\hat U_n(\hat U_n - \gamma \hat V_n)^T)$ is positive-definite. We make this matrix appear at different places through reformulations, e.g., applying the Woodbury identity to $Q_m(\lambda)$ modifies its structure, enabling us to leverage the symmetric/skew-symmetric decomposition of the adjusted matrix.

Reviewer comment (Evaluation):
I wonder if the observed double-descent behavior may be specifically due to the analysis of MSBE rather than some other learning objective, like the mean-squared value error (MSVE), the error relative to $v_\pi$. For example, it is known that TD methods that optimize MSPBE can converge to a solution far away from the minimum MSBE, which may explain the large peak around $N = m$. However, if we measure against MSVE instead, would we still see this? I imagine that if we have a set of features with some MSVE, and then add more features, we could just set those corresponding new weights to zero to preserve the MSVE. This seems to imply to me that the MSVE would be monotonically non-increasing as $N$ increases, although I am not sure if I am missing something. Perhaps the authors could better justify their choice of MSBE in this regard.

Author response:
You raise the question of whether double descent also occurs when learning objective functions other than the MSBE. We have considered this problem in detail and added both theoretical and experimental results to the appendix. Additional experiments in Appendix B indicate that the double descent phenomenon is also observed for the Mean-Square Value Error (MSVE). With a proof similar to that for Theorem 3, we derive a deterministic limit for the MSVE, as detailed in Appendix B. We focused on the MSBE for our study because both the MSBE and the MSPBE converge when $N \to \infty$ and since the MSPBE is the true learning objective function used by TD learning algorithms.

Reviewer comment (Experiments):
The paper says that only one random seed (42) was used to generate the training data, so I have to guess that only one seed was used to generate the random features as well. Since the MSBE is highly dependent on these “network” features, it is impossible to tell from just one trial that the double-descent observation is a common phenomenon over the distribution of possible feature initializations. It would be helpful to show the results averaged over a number of different networks, with confidence intervals shown.

Author response:
Thank you for highlighting the lack of detail regarding the seeds used for generating random features. We have provided additional details in the description of the experimental section in order to clarify this issue. Each curve in the experiments represents the average over 30 training runs. A different seed is used to generate the random features in each training run. 
For the sake of clarity in the presentation, we chose not to include standard deviations in the figure of the experimental section, but they are illustrated in Figure 1.

Reviewer comment (Theory):
Can the norm term in $\bar{\text{MSBE}}(\hat{\theta})$ (Theorem 5.3) be simplified to $\|| \bar{r} + (\gamma P - I) \frac{1}{\sqrt{n}} \frac{N}{m} \frac{1}{1+\delta} \Phi_S U_n \bar{Q}_m (\lambda) r \||^2_D$, i.e., factoring out $\gamma P - I$? Or was it left expanded intentionally?

Author response:
The factorization is correct, but we have opted for the expanded form in order to explicitly indicate that it represents an MSBE and for visual consistency with the other definitions of MSBE throughout the paper.

Reviewer comment (Experiments):
Why was recursive LSTD used for the experiments? Since only the final weights are needed, would it be more efficient to just sample data from the behavior policy and then explicitly calculate the TD fixed point like $w = A^{-1} b$?

Author response:
We use the Recursive LSTD because the classic LSTD requires more computational resources to process large datasets, particularly for inverting large matrices. Although Recursive LSTD may introduce a greater potential for instability, our experimental results are consistent with the theoretical expectations.