Paper: Vlearn: Off-Policy Learning with Efficient State-Value Function Estimation (ICLR-2024)

Reviewer comment (Theory):
Lack of mathematical rigor in the algorithm description (see questions below)

Author response:
- **Derivation Details and Theoretical Justification:** We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound.
We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

Reviewer comment (Writing):
In eqn (2) and the equation right above eqn (4): what is \( j \), what is the sum over \( j \), and what is \( K \)? Below eqn (2), the text says "\( a_{t,j} \) defines the \( j \)th action in state \( s_t \)". Does this mean \( j \) is enumerating the collected transitions (presumably in a buffer)? Or is this following some other distribution — what distribution would that be?

Author response:
To answer some of the questions already at this point. The $\sum_j^K$ over actions in state $s_t$ is a Monte Carlo estimate of the expectation over actions from our policy. In practice computing this is hardly possible as we would have to be able to execute multiple actions from the same state. Our method now moves this expectation outside of the square and allows us to use the joint expectation over states and actions. But we agree this should be also reflected in the notation, we updated it accordingly.

Reviewer comment (Theory):
Right above eqn (3), clarification is needed on what "upper bound" and "lower bound" refer to in that sentence, and why "truncated importance weights can provide an upper bound".

Author response:
- **"Upper Bound" and "Lower Bound" Clarification Eq (3):** The upper bound here simply refers to the truncated importance weights that clip the importance weights at typically 1. While we do not leverage a specific lower bound, the importance ratio has a natural minimum at 0.

Reviewer comment (Novelty):
In Mahmood et al. 2014, the authors considered the linear function approximation setting and presented two importance sampling approaches for off-policy learning. Their OIS-LS and WIS-LS seem to correspond to the difference between eqn (2) vs eqn (4) of this paper. Specifically, their WIS-LS objective is \( J(\theta) = \frac{1}{n} \sum_k \rho_k (Y - \theta^{\intercal} \phi_k) \), which basically corresponds to eqn (4) of this paper \( L(\theta) = \frac{1}{K} \sum_{t} \rho_t (V_\theta(s_t) - (r_t + \gamma V_{\bar{\theta}}(s_{t+1})))^2 \) except now \( V_\theta \) is used in place of a linear function.

Author response:
- **Contribution and Connection to Mahmood et al. 2014:** Our work is indeed similar to this previous work and could be seen as an extension to the general case of function approximation. However, direct application of this solution is still challenging as seen in our added ablation study, hence we also propose a proper way on how to leverage these results in an efficient way as part of an off-policy trust region method. Although a direct application is not possible, we believe that this related work further supports our argument that using importance sampling for the full loss function is preferable. We have made this connection clearer in the revised version.

Reviewer comment (Novelty):
The core idea of the current proposed approach seems to be simply extending the linear case in Mahmood et al. 2014 to general function approximation. The reviewer asks whether there is any other difference and suggests that this connection should be explicitly stated, perhaps citing the mathematical formulation as well.

Author response:
- **Contribution and Connection to Mahmood et al. 2014:** Our work is indeed similar to this previous work and could be seen as an extension to the general case of function approximation. However, direct application of this solution is still challenging as seen in our added ablation study, hence we also propose a proper way on how to leverage these results in an efficient way as part of an off-policy trust region method. Although a direct application is not possible, we believe that this related work further supports our argument that using importance sampling for the full loss function is preferable. We have made this connection clearer in the revised version.

Reviewer comment (Writing):
Minor typo on page 4 paragraph 2: "Bellmann" → "Bellman"

Author response:
- **Minor Issues:** Thank you for pointing out the typographical errors. In the revised manuscript, we corrected these inaccuracies.

Reviewer comment (Writing):
Minor typo on page 6: "populate]"

Author response:
- **Minor Issues:** Thank you for pointing out the typographical errors. In the revised manuscript, we corrected these inaccuracies.

Reviewer comment (Theory):
The deduction of the new surrogate objective for the value function learning lacks important explanations. Specifically, regarding the statement “accomplish this by relaxing the squared loss function from Equation 2 using Jensen’s inequality”, it is not straightforward how to use Jensen’s inequality to obtain \(L(\theta)\), and why the ratios can be moved just outside of the square.

Author response:
**Derivation Details and Theoretical Justification:** We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound.
We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

Reviewer comment (Theory):
In Equation (4), the paper subsumes the summation over \(j\) into the sum over \(t\), but it is unclear how this can be done given that the summation over \(j\) “implicitly assumes multiple action executions per state”. The \(L(\theta)\) in Equation (4) clearly does not have such an implicit assumption. Furthermore, there is no discussion of this in subsequent sections, and Algorithm 1 only presents a single-action estimate for the loss.

Author response:
**Derivation Details and Theoretical Justification:** We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound.
We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

  To already address the question regarding the summation at this point, we indeed use a one sample estimate (as does V-trace), the formulation was probably misleading there. By moving the importance sampling out of the square in Vlearn, it allows us to work with the joint expectation over states and actions (this is what we referred to as subsummation). This way having only one action sample per state is less of an issue and leads to lower variance than having a one sample Monte Carlo estimate just for the expectation over actions, such as in V-trace. This is similar to what SAC is doing by only generating one action sample for its expectation over the Q-function. V-trace effectively splits the expectations of states and actions and still uses a one sample estimate for the expectation over actions in Eq. (2), leading to potentially higher variance.

Reviewer comment (Theory):
The conclusion from the cited work (Neumann & Peters, 2008) that “the new loss function is an upper bound of the original loss function and shares the same optima” may not carry over straightaway to the value estimation in this paper. In the cited work, the proof is for the value estimation of \(\pi\) with a given Q and samples generated by the same policy. This differs from the objective function in Equation (4), which is estimated using target networks and off-policy samples. The paper should provide a theoretical proof for convergence to the same optima. Additionally, the meaning of “solving the relaxed loss should yield results consistent with the original loss” is unclear—whether it refers to consistency of the final loss or of the final policy.

Author response:
In response to your initial review, we have included a detailed derivation of the upper bound, a consistency proof based on Neumann & Peters, 2008, tailored to our specific case, and an extensive ablation study. This study highlights the impact and benefits of various elements of our final proposed method, for example, showing the performance differences between using the exact same algorithmic setup but with V-trace or without importance sampling.

However, we recognize that our interpretation may not have fully met your expectations. Could you please provide further guidance on the specific details or aspects you were expecting? 

Thank you very much for your time and consideration.

Reviewer comment (Experiments):
It is unclear how the proposed state-value estimation can be considered useful for the algorithm. The proposed algorithm combines multiple independent ideas—state value estimation, target networks, trust region projection, Gaussian policy approximation, replay buffer, and importance weight truncation—any of which could be crucial for the empirical performance. It is hard to isolate the effect of the state value estimation and claim it is the key contributor.

Author response:
**Utility of State-Value Estimation:** We agree with the reviewer that the effects of the individual components are currently difficult to filter out. We have extended the ablation section of the paper by performing additional variations of our proposed method. However, it should already be noted here that in our experiments we already compare to V-trace in an equivalent setting, using the same techniques and only changing the way the value function is trained. Nevertheless, V-trace is not able to achieve the same performance. For this reason, we also think that a comparison with a Monte Carlo estimator is difficult, since V-trace is not able to learn a reasonable policy, which would also be reflected in a relatively poor estimate of V-trace in this comparison.

Reviewer comment (Experiments):
Further empirical studies are necessary to verify the proposed idea. The current empirical results only present evaluation reward/returns for the trained policies, which are insufficient to show that the proposed value estimation objective improves the accuracy of value estimation. The strong performance of Vlean could be due to other implementation tricks. The paper should consider using Monte Carlo estimates to compare estimated values from Vlean to ground truth values. Furthermore, the ablation studies on replay buffer size are irrelevant to the proposed value estimation and should be replaced with more informative studies, such as testing with/without the ratios when optimizing \(L(\theta)\). Current ablation studies do not clarify the effect on state-value-function estimation.

Author response:
In response to your initial review, we have included a detailed derivation of the upper bound, a consistency proof based on Neumann & Peters, 2008, tailored to our specific case, and an extensive ablation study. This study highlights the impact and benefits of various elements of our final proposed method, for example, showing the performance differences between using the exact same algorithmic setup but with V-trace or without importance sampling.

However, we recognize that our interpretation may not have fully met your expectations. Could you please provide further guidance on the specific details or aspects you were expecting? 

Thank you very much for your time and consideration.

Reviewer comment (Writing):
There is a minor language issue: “This loss from Equation 5 can be optimized” is inaccurate because Equation 5 is not a loss but an objective to maximize.

Author response:
**Language & Clarity:** We acknowledge there are discrepancies between Algorithm 1 and the method as stated in the main text. In the revised manuscript, we improved the algorithm description to make it more self-explanatory and aligned. We also fixed the mentioned language issues.

Reviewer comment (Theory):
Why does the learning objective in Vlearn work better than 1-step V-trace is still not clear to me. Despite the empirical results, it will be more sound if the author could give some (theoritical) analysis to it.

Author response:
- **Comparison to V-trace:** By moving the importance sampling out of the square in Vlearn, it allows us to compute the joint expectation over states and actions. This way having only one action sample per state should generally lead to less variance. V-trace effectively keeps the split expectations of states and actions and still uses a single sample to estimate the expectation over actions in Eq. (2), leading to potentially higher variance. Furthermore, V-trace is interpolating between the current (target) value function and the 1-step Bellman target
$$
    J(\theta) = E_{s_t \sim \mathcal{D}} \Bigg[ \Big( V_\theta^\pi(s_t) - 
    \Big(
    (1-\rho_t) V_{\bar{\theta}}(s_t) + \rho_t \big(r_{t}+ \gamma V_{\bar{\theta}}(s_{t+1}) 
    \big)
    \Big)
    \Big)^2
    \Bigg].
$$
The smaller the importance ratio, the more the value function optimization focuses on the current (target) value function. However, for these samples, the overall loss scale and consequently the gradient remain unchanged. In the case of Vlearn, it is different – samples with small importance ratios contribute less to the total loss but still work towards the Bellman target. V-trace was initially designed for correcting off-policy asynchronous processing, where policy differences (i.e. importance ratios) are typically limited, so this is not much of a constraint. However, in a complete off-policy scenario, the samples in the replay buffer and the current policy deviate more rapidly and significantly from each other. We tried to emphasize this insight more in the revised version to make the distinction clearer.

Reviewer comment (Evaluation):
Empirical results on easy, low-dimension task such as hopper, walker, cheetah are low.

Author response:
- **Low-Dimensional Tasks:** We agree that our performance does not provide significant improvements for the low-dimensional task, but we did not expect it to, since the added complexity of learning a Q-function is likely negligible in these cases. Therefore, similar performance is a desirable result for our method. SAC on the HalfCheetah-v4, we see as an outlier, as other off-policy methods, such as MPO, also do not perform as well on this task, and it has been shown in previous work [1] that this particular task exhibits extreme behavior.

Reviewer comment (Theory):
The derivation skips directly to the final equation, not providing enough detail. It's also possibly erroneous. Please see the question section.

Author response:
**Derivation Details and Theoretical Justification:** We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound.
We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

Reviewer comment (Experiments):
The ablation study only investigates the effect of the size of the replay buffer. However, Vlearn uses multiple techniques from previous works, e.g., importance sampling ratio truncation. Each of them may contribute to the reported final performance. A more comprehensive ablation study can definitely enhance the value of this work.

Author response:
**Effect of Individual Components:** We agree with the reviewer that an ablation study on the effects of the individual components can be of good value. We extended this section of the paper by running additional experiments. However it should already be noted here that in our experiments we compare to V-trace in an equivalent setting using the same techniques and only changing the way the value function is trained. Yet, V-trace is not able to achieve the same performance.

Reviewer comment (Writing):
Some notations are used before definition. For example, I was unsure what $K$ represents until I got to the pseudocode to find out it was the mini-batch size.

Author response:
**Derivation Details and Theoretical Justification:** We acknowledge your concern about the ambiguity in the derivation of the upper bound for the value function. In the revised manuscript, we have improved the notation in the main text and provided a comprehensive derivation in the appendix. In addition, we have given more insight into the method, explaining its relationship to and differences from V-trace. We have also now included a full proof establishing the consistency of the upper bound.
We hope that these revisions not only answer your questions about the derivation, but also provide a more intuitive understanding of the method. Should you require further clarification, we are available to provide additional information.

Reviewer comment (Reproducibility):
The pseudocode is somewhat vague. For instance, does the for $ i \in \\{1,2\\}$ indicate a for-loop or a random sample from the set in lines 9 and 13? What does $\theta$ in line 11 represent? It's not in the input to the algorithm.

Author response:
**Pseudocode:** Here, the "for" is just an indicator that we compute this loss for both critic networks and equivalently for the target critics in line 13, i.e. a for loop over critics. We decided on this notation since it is also commonly used in other works, e.g. SAC [1]. The $\theta$ in line 11 is indeed inconsistent, and should be the minimum of the two critic estimates. Overall, for the updated version, we have aligned the pseudocode with the main text.

Reviewer comment (Theory):
What is the motivation behind using two sets of parameters for the state-value function? The overestimation bias exists in Q-learning. I don't think it exists in the authors' formulation.

Author response:
**Overestimation Bias:** We agree the overestimation bias is not a main concern in our setting as compared to the Q-function based setting. We found, however, in practice that using two networks is also beneficial in our case (see new ablation study). We assume the second network can be seen as a small ensemble that acts as a form of regularization and by selecting the minimum value, we avoid introducing any overestimation bias by training two networks. As the twin network approach is common practice for the baseline methods, we kept it also in our case.