Paper: Compound Returns Reduce Variance in Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
I think that most of the results are slightly incremental and not clearly novel.

Author response:
Our work is the first to make progress on this question in the last 12 years. Please see “**Strength of Theoretical Contributions**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=TJYI9LMG6P) for more details about the novelty of our theory.

Reviewer comment (Theory):
In general in RL it is not clear if minimizing the variance of the return estimators is helpful to improve the sample complexity of an algorithm. Check for example this paper investigating the role of the minimum variance baseline in policy gradient as in [1].

Author response:
This is an excellent point. It is true that reducing the return variance is not guaranteed to improve the sample efficiency of *policy gradient* methods. The paper that you cited shows that the variance of the *baseline* can sometimes be beneficial because of its interaction with the exploration policy (what the authors call committal and non-committal baselines). Our value-based setting is fundamentally different because it does not rely on a critic to determine which actions should be reinforced. Instead, the policy is directly determined as a function of the value function (e.g., $\epsilon$-greedy) and so the main bottleneck for learning is how quickly the returns can be estimated accurately.

Furthermore, it is widely accepted that variance reduction is beneficial for *learning value functions*. This is why TD-based returns (e.g., n-step returns [2] or lambda-returns [3]) are almost always used in deep RL instead of Monte Carlo returns, as the latter have extremely high variance. To isolate the effects of return estimation itself, we chose to focus on DQN, a well-studied deep RL method that only learns a value function. The main benefit of our method, PiLaR, is to apply similar variance reduction in off-policy experience-replay settings where computing the full $\lambda$-return is infeasible.

Reviewer comment (Experiments):
The experiments in Deep RL are limited to only one environment. I think that a larger empirical evaluation is necessary.

Author response:
Thank you for this suggestion. We have added results for more MinAtar games to the paper; please see “**Additional Experiments**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=GK2F7B8JpD).

Reviewer comment (Theory):
Is it possible to use the results in this paper to show more informative results regarding the performance of TD($\lambda$)? For example, that having a lower variance in the returns improves the sample complexity needed for either policy evaluation or for learning an $\epsilon$-optimal policy?

Author response:
This is a great question. Please see “**Why Variance Reduction Leads to Faster Learning**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=GK2F7B8JpD) for a detailed discussion of how the contraction rate corresponds to expected policy improvement in Q-Learning methods. We have also added this discussion to our paper’s appendix.

Reviewer comment (Experiments):
It is unclear whether the uniform covariance assumption is reasonable in real-world problems, since the hardness to approximate the covariance between different steps should not be an evidence to support the validity of this assumption. Intuitively, the variance of TD errors at further steps should be larger since the entropy of state should increase along the diffusion over the MDP. Therefore, it is appreciated to verify this assumption empirically on synthetic examples.

Author response:
Our work is the first to make progress on this question in the last 12 years. Please see “**Strength of Theoretical Contributions**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=TJYI9LMG6P) for a detailed discussion.

We think it is a great idea to measure how well our assumptions hold in practice. We added two experiments to the paper (Appendix B) that plots the actual variance of the n-step returns compared to the lower and upper bounds predicted by our model.

Reviewer comment (Evaluation):
The contraction rate measures the contraction level of the policy evaluation process. It is not clear the effect of this rate in the policy optimization process, nor is it discussed in the paper. Therefore, it is still not clear whether the faster learning with DQN is a consequence of smaller variance of PiLaR or smaller contraction rate in the policy optimization process as $n_2$ is generally larger than $n$.

Author response:
Thank you for bringing up this point. Please see “**Why Variance Reduction Leads to Faster Learning**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=GK2F7B8JpD) for a detailed discussion of how the contraction rate corresponds to expected policy improvement in Q-Learning methods.

Furthermore, it can be seen from the MinAtar experiments (Figures 5 and 6) that a larger value of $n$ actually hurts performance due to the increased variance, so it is not true that a larger $n_2$ value for PiLaR is responsible for the faster learning. Since each pair of returns are chosen to have the same contraction rate, the only remaining explanation is the reduced variance, which is supported both by our theory and random walk experiments in Figure 1.

Reviewer comment (Theory):
The theoretical results of the paper are mostly conceptual in the sense that it proves some variance reduction results but do not discuss how this lower variance accelerate the learning of optimal policies. The "equally fast" claim for two estimators with the same contraction rate is also conceptual without solid evidence. Does it correspond to smaller sample complexity in theory? The insight of this paper is also limited in both practice and theory, since the baseline is the $n$-step TD learning and DQN, which is away from current SOTA algorithms used in RL. Is is possible to compare the PiLaR (or more refined compound error with even smaller variances) with some SOTA RL algorithms such PPO or CQL?

Author response:
This is a great question. We would first like to point out that our random walk experiment in Figure 1 does provide solid evidence that two returns with the same contraction rate learn equally fast in expectation (small step size) but the one with lower variance learns faster with limited samples (large step size). Again, please see “**Why Variance Reduction Leads to Faster Learning**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=GK2F7B8JpD) for answers to your questions regarding contraction rate and sample complexity.

The principal contribution of our paper is the theory. The purpose of our experiments is to test how well our theory is reflected in practice when the assumptions required for the math do not always hold, as well as to demonstrate the feasibility of using variance-reducing compound returns in deep RL. We further discuss this in “**Additional Experiments**” in our [general response](https://openreview.net/forum?id=f37TVPH62h&noteId=GK2F7B8JpD), where we provide results for more MinAtar games.

Studying actor-critic methods such as PPO or CQL would add a confounding factor since they simultaneously train a stochastic policy (as you correctly noted above). To isolate the effects of return estimation itself, we chose to focus on DQN, a well-studied deep RL method that only learns a value function.

We also note that PPO is an on-policy method and commonly uses the $\lambda$-return with GAE [1] rather than the high-variance Monte Carlo return, which provides further evidence that variance reduction is beneficial in practice. Thus, the main benefit of PiLaR is to apply similar variance reduction in off-policy experience-replay settings where computing the full $\lambda$-return is infeasible.

**References**

[1] High-dimensional Continuous Control Using Generalized Advantage Estimation. John Schulman et al., ICLR 2016.