Paper: Policy Gradient without Boostrapping via Truncated Value Learning (ICLR-2024)

Reviewer comment (Experiments):
The motivation need to be further addressed. The authors demonstrate the advantage of horizon truncation in section 3, showing that untruncated TD update can lead to divergence. However, one weakness of this demonstration is the authors set h_max to be 1000, which is the effective horizon of $\gamma=0.999$. With this hard constraint, further increasing $\gamma$ has very little effect on the convergence result because they are "cut off" back to $0.999$ due to the constraint (as one can see from the figure, the slope is much smaller when $\gamma$ exceeds 0.999). In this case, the comparison for $\gamma > 0.999$ is not fair. On the other hand, when one looks the part of $\gamma < 0.999$, both curves show good convergence results, which makes the authors' claim less convincing.

Author response:
Your point regarding the setting of h_max in the toy example is a good one. We set h_max to 1000 as this was the effective horizon of the first diverging TD case (gamma=0.999). However, it would have been better (for the reasons you have mentioned) to extend this beyond the effective horizon. We therefore have rerun our experiment with h_max = 10,000​ and found that Fixed Horizon (FH) updates did not diverge in either the 0.999 or 0.9999 gamma scenarios (for gamma=1.0, the error is unchanged from initialization). We look forward to updating our paper with these results, and they you very much for drawing our attention to this limitation. We believe this new data will more convincingly establish our claim regarding the divergence of TD in settings with high noise and high gamma, where FH does not. With the revised h_max, the impact of future rewards, even beyond this point, becomes negligible for a gamma of 0.999. This update should, hopefully, address your concerns by setting the horizon limit well above the effective horizon for our gamma values.

We also agree that setting h_max low would cause the algorithm to perform well on this toy task (but not others) because it would essentially be applying heavy discounting. However, in this case, we are not concerned with the accuracy of the prediction so much as if the errors cause instability during training. Our intention is to show that even with very long horizons, TD still diverges but that this is not the case for FH.

Reviewer comment (Experiments):
One would like to see what happens for a larger h_max value when testing the convergence result for a larger $\gamma$ value, for example, one can set h_max=10000 for $\gamma=0.9999$ so that no long term signals are killed.

Author response:
Your point regarding the setting of h_max in the toy example is a good one. We set h_max to 1000 as this was the effective horizon of the first diverging TD case (gamma=0.999). However, it would have been better (for the reasons you have mentioned) to extend this beyond the effective horizon. We therefore have rerun our experiment with h_max = 10,000​ and found that Fixed Horizon (FH) updates did not diverge in either the 0.999 or 0.9999 gamma scenarios (for gamma=1.0, the error is unchanged from initialization). We look forward to updating our paper with these results, and they you very much for drawing our attention to this limitation. We believe this new data will more convincingly establish our claim regarding the divergence of TD in settings with high noise and high gamma, where FH does not. With the revised h_max, the impact of future rewards, even beyond this point, becomes negligible for a gamma of 0.999. This update should, hopefully, address your concerns by setting the horizon limit well above the effective horizon for our gamma values.

We also agree that setting h_max low would cause the algorithm to perform well on this toy task (but not others) because it would essentially be applying heavy discounting. However, in this case, we are not concerned with the accuracy of the prediction so much as if the errors cause instability during training. Our intention is to show that even with very long horizons, TD still diverges but that this is not the case for FH.

Reviewer comment (Theory):
While the algorithm learns the h-horizon values for several horizons, when combined the value function with PPO, one only finds the appearance of the value function for h_max (for example, equation 12). This is somehow disappointing. It could be possible that the value estimations for various horizons are hidden somewhere in this expression like the TD updater. But if that is the case, the authors should address it in the main context as this is crucial for showing why their formulation is useful.

Author response:
We agree that exploring learned horizons could be advantageous. In earlier stages, we experimented with dynamically modifying the policy discount rate, effectively choosing shorter horizons from the learned value function. However, we found that opting for higher horizons was (almost) always beneficial. The crucial aspect of Trunched Value Learning (TVL) lies in its robust value learning, which remains stable even in noisy settings. After these estimates are learned, the final estimate is the one that matters, but the learning process necessitates the full range.

Reviewer comment (Evaluation):
Part of the experiments didn't show convincing results. For both General Performance and Long Horizon, the algorithm doesn't significantly outperforms the baseline.

Author response:
We consider state-of-the-art results on the challenging Procgen benchmark a significant result. We were disappointed that TVL did not work well for long horizons, but we believe strongly that negative results should be published. This is an incremental step forward in an important area, and we show, at the very least, that the full horizon curve can be learned and that it improves performance in the challenging high-noise Procgen environment, which we believe would be of interest to the readers.

Reviewer comment (Theory):
What is the justification for using linear interpolation to estimate the value function of an unknown horizon $h$?

Author response:
Why linear-interpolation? In earlier versions of the algorithm, we modelled the value curve directly, i.e. used a DNN to perform interpolation. This worked quite well but was overly complicated. We later discovered that linear interpolation worked just as well but was *much* simpler to implement, and we went for the simplest approach. Linear interpolation does have disadvantages. For example, if the environment was aperiodic, we might have an issue with rewards always appearing between the fixed head locations. We did not find this to be an issue in the environments tested, but we would be happy to write up this limitation if accepted.

Reviewer comment (Experiments):
How did the authors choose the number of horizons K? If one uses a larger K, will one always expect a better performance so that the only constraint is computation resource? Or there is already some trade-off over the statistical performance so that increasing K can actually decrease the estimation accuracy / cumulative reward at some time point?

Author response:
Number of heads? We found the algorithm to be robust to the parameter K. We did some initial hyperparameter searches and found 128 to be a good tradeoff between compute performance and the quality of the value estimates. We would be happy to include an ablation study on the values of K in an updated version of our paper for publication. There is a tradeoff in the quality of the value estimates (along the horizon curve) when using higher / lower K. However, this does not necessarily translate into large differences in performance (unless K is very low). The algorithm works quite well even with a coarse value curve representation.

Reviewer comment (Experiments):
This paper lacks enough ablation experiments, making it difficult to see the role of each component clearly. For example, the author should have shown the benefits of TVL not using traditional discounting while learning rewards.

Author response:
Thank you for your thorough review and valuable feedback.

We agree with your suggestion that incorporating an ablation study would enhance the overall quality of our paper. We are planning to include an ablation study in the revised version of the paper, subject to its acceptance. This study will specifically examine the sensitivity of the newly introduced hyperparameters, and will add this study to the appendix. Our observations during the development of the Truncated Value Learning (TVL) algorithm indicated a robust performance across a wide range of hyperparameters, a detail that our current manuscript may not have adequately highlighted. We are looking forward to addressing this in the revised paper.

Reviewer comment (Evaluation):
TVL has yet to be significantly proposed compared with the baseline algorithm on the long-horizon task, so it is difficult to judge whether the TVL method is effective on the long-horizon task.

Author response:
In terms of comparing our algorithm to others in long-horizon scenarios, we believe our paper already presents these results. Specifically, we found that TVL does not surpass DNA in long-horizon tasks. We chose to use the Skiing game for evaluation over Pitfall, as Skiing presents a different challenge: rewards are deferred rather than sparse. This distinction allows for more informative rewards per episode compared to Pitfall, where most trajectories yield uninformative rewards. Where we did find improvement was in TVL's performance in a high-noise setting. Our interest in Skiing was due to it being the most challenging game for Agent57, which only solved it by increasing the discount to 0.9999. We believe this is a strong indicator that skiing is a good choice for long-horizon evaluation. While other games like Montezuma's Revenge and Pitfall also represent long-horizon challenges, they involve additional complexities related to hard exploration, an area our algorithm does not specifically aim to enhance.

Reviewer comment (Experiments):
The author may be able to test the proposed method on more long-horizon tasks to increase persuasiveness, such as pitfall, etc.

Author response:
In terms of comparing our algorithm to others in long-horizon scenarios, we believe our paper already presents these results. Specifically, we found that TVL does not surpass DNA in long-horizon tasks. We chose to use the Skiing game for evaluation over Pitfall, as Skiing presents a different challenge: rewards are deferred rather than sparse. This distinction allows for more informative rewards per episode compared to Pitfall, where most trajectories yield uninformative rewards. Where we did find improvement was in TVL's performance in a high-noise setting. Our interest in Skiing was due to it being the most challenging game for Agent57, which only solved it by increasing the discount to 0.9999. We believe this is a strong indicator that skiing is a good choice for long-horizon evaluation. While other games like Montezuma's Revenge and Pitfall also represent long-horizon challenges, they involve additional complexities related to hard exploration, an area our algorithm does not specifically aim to enhance.

Reviewer comment (Experiments):
Can the authors add some ablation experiments to supplement the paper?

Author response:
Thank you for your thorough review and valuable feedback.

We agree with your suggestion that incorporating an ablation study would enhance the overall quality of our paper. We are planning to include an ablation study in the revised version of the paper, subject to its acceptance. This study will specifically examine the sensitivity of the newly introduced hyperparameters, and will add this study to the appendix. Our observations during the development of the Truncated Value Learning (TVL) algorithm indicated a robust performance across a wide range of hyperparameters, a detail that our current manuscript may not have adequately highlighted. We are looking forward to addressing this in the revised paper.

Reviewer comment (Experiments):
There are many other long-horizon tasks in Atari, such as pitfall. Skiing alone is not convincing enough (and the performance in skiing does not seem to be significantly improved). Can the author add some results of other tasks? This might increase the convince of the proposed method.

Author response:
In terms of comparing our algorithm to others in long-horizon scenarios, we believe our paper already presents these results. Specifically, we found that TVL does not surpass DNA in long-horizon tasks. We chose to use the Skiing game for evaluation over Pitfall, as Skiing presents a different challenge: rewards are deferred rather than sparse. This distinction allows for more informative rewards per episode compared to Pitfall, where most trajectories yield uninformative rewards. Where we did find improvement was in TVL's performance in a high-noise setting. Our interest in Skiing was due to it being the most challenging game for Agent57, which only solved it by increasing the discount to 0.9999. We believe this is a strong indicator that skiing is a good choice for long-horizon evaluation. While other games like Montezuma's Revenge and Pitfall also represent long-horizon challenges, they involve additional complexities related to hard exploration, an area our algorithm does not specifically aim to enhance.

Reviewer comment (Evaluation):
Long-horizon brings the higher sample inefficient for online algorithms. The training time is 3x higher than the PPO algorithm.

Author response:
Regarding the performance aspect of our algorithm, we acknowledge that it is three times slower than traditional approaches like PPO. However, this comparative decrease in speed represents a notable advancement compared to algorithms that scale linearly for large values of h. This sub-linear scalability is a significant attribute, demonstrating the robustness and potential of our method in handling very long time horizons, which is a crucial factor in complex environments and tasks.

Reviewer comment (Experiments):
This paper involves multiple hyperparameters, e.g., $k, c_{vh}$, without sufficient ablation study.

Author response:
We recognize that the Truncated Value Learning (TVL) approach introduces additional hyperparameters. However, our findings suggest that the algorithm is quite robust to these parameters across the tested environments. Notably, TVL eliminates the need to fine-tune the gamma parameter per environment, as it performs consistently well with a high gamma across tasks. To further validate this, we are willing to include an ablation study in the appendix upon acceptance of the paper, demonstrating the sensitivity of the new hyperparameters introduced. In practice, we observed that TVL requires less tuning than algorithms like PPO or DNA, owing to its independence from environment-specific gamma adjustments.

Reviewer comment (Writing):
The lack of citation in the first time mentioning DNA algorithm.

Author response:
Thank you for your valuable observations regarding our paper, including the need to cite the DNA reference and update our notation for the policy and value network parameters theta. We appreciate these insights and will incorporate them into a revised version of our paper.

Reviewer comment (Writing):
The parameters in different networks should not be all $\theta$, try to use different notations for different networks.

Author response:
Thank you for your valuable observations regarding our paper, including the need to cite the DNA reference and update our notation for the policy and value network parameters theta. We appreciate these insights and will incorporate them into a revised version of our paper.

Reviewer comment (Writing):
It is hard for me to follow to distillation part. For example, what is $c_{vh}$, and why we should use distillation? What is actually doing in Eq. 14?

Author response:
You have made a good point regarding the level of detail we included on the distillation process. We also plan to provide more detailed insights into the distillation process, recognizing its complexity and importance. To make our paper more self-contained, we will include a brief summary of the DNA distillation process.