Paper: ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update (ICLR-2024)

Reviewer comment (Theory):
Theorem 3: Given that you measure feature co-adaption with an un-normalized dot-product, couldn't the result in Theorem 3 be achieved by a shrinking of the feature vectors rather than a change in their orientation or relative representations? I assume your intention is to show that $\nabla_{\theta} V(s)$ and $\nabla_{\theta} V(s')$ are more different (or separated) under the orthogonal-gradient update (i.e., $\theta=\theta''$) than they are normally (i.e., $\theta=\theta'$).

Author response:
We acknowledge that shrinking the feature vectors can, to some extent, decrease $\Psi_\theta(s, s')$. However, without introducing an auxiliary optimization objective to DICE methods, $\| \nabla_\theta V(s) \|$ is often bounded, as assumed in Theorem 3 with $m<| \nabla_\theta V(s) |<M$. In fact, when employing practical tricks for training stability, such as layer normalization[1], $\| \nabla_\theta V(s) \|$ is even confined to a very small interval. Therefore, a smaller $\Psi_\theta(s, s')$ typically indicates that feature vectors are more orthogonal in orientation rather than simply having a smaller norm.

[1]Layer normalization. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.

Reviewer comment (Presentation):
Figure 2: I can see how (d) aligns well with the dataset support, but so does (b). Is (b) undesireable because the actual values V(s) across the offline states are poorly approximated (i.e., the gradient across V(s) is uninformative)? If so, can you please add that to the caption?

Author response:
We apologize for not including this in the caption. You are correct, and we will add it in the updated version.

Reviewer comment (Writing):
You say "Hence, Eq.(36) < 0" above Theorem 4? Is this a typo? Did you mean to refer to equation 6?

Author response:
We are sorry that we mistakenly wrote "Eq.(6)" as "Eq.(36)" here and it will be addressed in the updated version.

Reviewer comment (Theory):
The theoretical motivations for orthogonal gradient update do not seem to be sufficient. What we can know with the theoretical results presented are: (a) If we put right \eta, the orthogonal gradient can be no worse than semi-gradient; (b) There is a possibility that orthogonal gradient can help feature co-adaptation; (c) Based on these, O-DICE should perform on par with S-DICE on simple enough domains, and on complex domains, using DR3-like regularizations will make S-DICE perform on par with O-DICE. Will there be additional source of better performance? Can we get better fixed point when orthogonal gradient with large \eta is adopted (large enough to make it different from S-DICE)?

Author response:
>The theoretical motivations for orthogonal gradient update do not seem to be sufficient. What we can know with the theoretical results presented are:
If we put right \eta, the orthogonal gradient can be no worse than semi-gradient
There is a possibility that orthogonal gradient can help feature co-adaptation
Based on these, O-DICE should perform on par with S-DICE on simple enough domains, and on complex domains, using DR3-like regularizations will make S-DICE to perform on par with O-DICE. Will there be additional source of better performance? Can we get better fixed point when orthogonal gradient with large \eta is adopted (large enough to make it different from S-DICE)?

1. One important thing is that in Theorem 2, we proved that with appropriate $\eta$ (large enough), orthogonal-gradient update can make the overall loss objective monotonically decrease and get a better fixed point. This is not guaranteed by semi-gradient update since semi-gradient can lie in different half-plane than true-gradient. So orthogonal-gradient is actually better than semi-gradient from this perspective.
2. Although DR3-like regularization could make similar improvement on feature representations, introducing auxiliary loss can also change the original gradient flow of DICE methods and thus lose the convergence property above. 
3. As shown in Theorem 2, 3, 4, we have additional better performance comes only by just utilizing projected backward gradient to enhance state-level constraint while elegantly retaining convergence properties as true-gradient.

Reviewer comment (Experiments):
Basically the orthogonal gradient technique used in this paper is not limited to DICE algorithms but can be adopted on any deep RL algorithms. Can we get improvements on ordinary deep RL algorithms?

Author response:
>Questions:
Basically the orthogonal gradient technique used in this paper is not limited to DICE algorithms but we can adopt them on any deep RL algorithms. Can we get improvements on ordinary deep RL algorithms?

We acknowledge that the orthogonal-gradient update could be applied to Bellman operator operations. However, a key factor contributing to the nice property brought by the orthogonal-gradient update in the DICE algorithm lies in the fact that the DICE algorithm does not minimize the traditional, **symmetric** MSBE (mean square Bellman loss), but instead minimizes an objective that incorporates a function $f^{\ast}$ that is **asymmetric**. As evident from Theorem 3, because in DICE $f^{*}$ is chosen to be monotonically increasing, $(f^\ast)'(x) \geq 0$ always holds so the RHS of Theorem 3 will less than 0. 
In other words, using orthogonal update in MSBE may not alleviate feature co-adaptation. Nevertheless, applying the orthogonal-gradient update to them may yield other intriguing properties, representing a potential avenue for future exploration.

Reviewer comment (Theory):
As far as I know, if we use the semi-gradient update for OptiDICE, it diverges since the second term dominates the first term, and the second term only increases the V function according to the monotonic increasing shape of f^*. It seems the trick of Sikchi et al. (2023) is the trick that makes the algorithm work. According to the objective in the paper, it seems the algorithm is sampling the arbitrary experience instead of initial state distribution, and it seems to be weighting it much more heavily. Is there any theoretical guarantee that the proposed objective function gives a similar solution to what we can get with OptiDICE?

Author response:
>As far as I know, if we use the semi-gradient update for OptiDICE, it diverges since the second term dominates the first term, and the second term only increases the V function according to the monotonic increasing shape of $f^*$. It seems the trick of Sikchi et al. (2023) is the trick that makes the algorithm to work. According to the objective in the paper, it seems the algorithm is sampling the arbitrary experience instead of initial state distribution, and it seems to be weighting it much more heavily. Is there any theoretical guarantee that the proposed objective function gives similar solution to what we can get with OptiDICE?
>OptiDICE actually tends to overestimate \nu, and similar to above, if S-DICE and O-DICE do not overestimate even without double Q trick, I believe that it should be related to the Sikchi et al. (2023) trick. is Sikchi et al. (2023) applied to OptiDICE in experiments?
>According to those reasons, I would like to see the results without the Sikchi et al. (2023) trick.

First we want to clarify that the trick is not first proposed by Sikchi et al. (2023). It is a common trick used in DICE paper such as ValueDICE and IQLearn to increase the diversity of samples. Theoretically, using a wider initial state distribution of offline dataset does not affect the policy performance at the environment's initial state distribution as long as the function-approximator has enough capacity (in our case they are neural networks) since the policy will only learn to achieve near-optimal behaviors from a wider variety of states.
Also, only O-DICE does not overestimate even without double Q trick, S-DICE still needs double Q trick to work. Suggesting the benefits come from using orthogonal update. The reason is that the gradient flow of $V$ can be reformulated as: (please refer to section 3.3)$$ (f^\ast)'(r+\gamma V(s') - V(s))\Big(\gamma\cdot\eta\nabla_\theta V(s') - \big(1 + \gamma\cdot \eta\frac{\nabla_\theta V(s)^\top \nabla_\theta V(s')}{\| \nabla_\theta V(s) \|^2}\big)  \nabla_\theta V(s) \Big) $$ Although $f^\ast$ is monotonic increasing, which would make $V(s_t)$ always increase or stay the same with the transition $(s_t, a_t, s_{t+1})$, $V(s_t)$ will decrease when we calculate gradient with the trasition $(s_{t-1}, a_{t-1}, s_t)$. Backward gradient will always make $V(s')$ decrease and thus remain undiverged.

To better see the benefits, we compare O-DICE with OptiDICE without the Sikchi et al. (2023) trick. It can be seen that the benefit clearly comes from using orthogonal update rather than using $d^D$.

|                 | O-DICE with $d_0$ | O-DICE with $d^{\mathcal{D}}$ | OptiDICE with $d_0$ | OptiDICE with $d^{\mathcal{D}}$ |
| --------------- | ----------------- | ----------------------------- | ------------------- | ------------------------------- |
| hopper-m        | 84.1              | 86.1                          |     49.9                |              46.4                   |
| hopper-m-r      | 99.4              | 99.9                          |      14.3               |               20.0                  |
| hopper-m-e      | 110.5             | 110.8                         |      58.1               |                51.3                 |
| walker-m        | 82.2              | 84.9                          |        61.2             |            68.1                     |
| walker-m-r      | 80.8              | 83.6                          |      17.8               |                17.9                 |
| walker-m-e      | 110.6             | 110.8                         |       101.3              |                104.0                 |
| halfcheetah-m   | 42.1              | 47.4                          |       42.0              |             45.8                    |
| halfcheetah-m-r | 43.2              | 44.0                          |        32.5             |             31.7                    |
| halfcheetah-m-e | 85.5              | 93.2                          |       53.0              |              59.7                   |

Reviewer comment (Experiments):
According to those reasons, I would like to see the results without the Sikchi et al. (2023) trick.

Author response:
>As far as I know, if we use the semi-gradient update for OptiDICE, it diverges since the second term dominates the first term, and the second term only increases the V function according to the monotonic increasing shape of $f^*$. It seems the trick of Sikchi et al. (2023) is the trick that makes the algorithm to work. According to the objective in the paper, it seems the algorithm is sampling the arbitrary experience instead of initial state distribution, and it seems to be weighting it much more heavily. Is there any theoretical guarantee that the proposed objective function gives similar solution to what we can get with OptiDICE?
>OptiDICE actually tends to overestimate \nu, and similar to above, if S-DICE and O-DICE do not overestimate even without double Q trick, I believe that it should be related to the Sikchi et al. (2023) trick. is Sikchi et al. (2023) applied to OptiDICE in experiments?
>According to those reasons, I would like to see the results without the Sikchi et al. (2023) trick.

First we want to clarify that the trick is not first proposed by Sikchi et al. (2023). It is a common trick used in DICE paper such as ValueDICE and IQLearn to increase the diversity of samples. Theoretically, using a wider initial state distribution of offline dataset does not affect the policy performance at the environment's initial state distribution as long as the function-approximator has enough capacity (in our case they are neural networks) since the policy will only learn to achieve near-optimal behaviors from a wider variety of states.
Also, only O-DICE does not overestimate even without double Q trick, S-DICE still needs double Q trick to work. Suggesting the benefits come from using orthogonal update. The reason is that the gradient flow of $V$ can be reformulated as: (please refer to section 3.3)$$ (f^\ast)'(r+\gamma V(s') - V(s))\Big(\gamma\cdot\eta\nabla_\theta V(s') - \big(1 + \gamma\cdot \eta\frac{\nabla_\theta V(s)^\top \nabla_\theta V(s')}{\| \nabla_\theta V(s) \|^2}\big)  \nabla_\theta V(s) \Big) $$ Although $f^\ast$ is monotonic increasing, which would make $V(s_t)$ always increase or stay the same with the transition $(s_t, a_t, s_{t+1})$, $V(s_t)$ will decrease when we calculate gradient with the trasition $(s_{t-1}, a_{t-1}, s_t)$. Backward gradient will always make $V(s')$ decrease and thus remain undiverged.

To better see the benefits, we compare O-DICE with OptiDICE without the Sikchi et al. (2023) trick. It can be seen that the benefit clearly comes from using orthogonal update rather than using $d^D$.

|                 | O-DICE with $d_0$ | O-DICE with $d^{\mathcal{D}}$ | OptiDICE with $d_0$ | OptiDICE with $d^{\mathcal{D}}$ |
| --------------- | ----------------- | ----------------------------- | ------------------- | ------------------------------- |
| hopper-m        | 84.1              | 86.1                          |     49.9                |              46.4                   |
| hopper-m-r      | 99.4              | 99.9                          |      14.3               |               20.0                  |
| hopper-m-e      | 110.5             | 110.8                         |      58.1               |                51.3                 |
| walker-m        | 82.2              | 84.9                          |        61.2             |            68.1                     |
| walker-m-r      | 80.8              | 83.6                          |      17.8               |                17.9                 |
| walker-m-e      | 110.6             | 110.8                         |       101.3              |                104.0                 |
| halfcheetah-m   | 42.1              | 47.4                          |       42.0              |             45.8                    |
| halfcheetah-m-r | 43.2              | 44.0                          |        32.5             |             31.7                    |
| halfcheetah-m-e | 85.5              | 93.2                          |       53.0              |              59.7                   |

Reviewer comment (Novelty):
Most RL algorithms involve Bellman operator operations, which require computing both forward and backward gradients. Therefore, orthogonal-gradient may not just be applicable to DICE-based methods, but also to most RL algorithms. This is much more important than improving DICE-based methods, and the author could consider this perspective.

Author response:
> Most RL algorithms involve Bellman operator operations, which require computing both forward and backward gradients. Therefore, orthogonal-gradient may not just be applicable to DICE-based methods, but also to most RL algorithms. This is much more important than improving DICE-based methods, and the author could consider this perspective.

We acknowledge that the orthogonal-gradient update could be applied to Bellman operator operations. However, a key factor contributing to the nice property brought by the orthogonal-gradient update in the DICE algorithm lies in the fact that the DICE algorithm does not minimize the traditional, **symmetric** MSBE (mean square Bellman loss), but instead minimizes an objective that incorporates a function $f^{\ast}$ that is **asymmetric**. As evident from Theorem 3, because in DICE $f^{*}$ is chosen to be monotonically increasing, $(f^\ast)'(x) \geq 0$ always holds so the RHS of Theorem 3 will less than 0. 
In other words, using orthogonal update in MSBE may not alleviate feature co-adaptation. Nevertheless, applying the orthogonal-gradient update to them may yield other intriguing properties, representing a potential avenue for future exploration.

Reviewer comment (Experiments):
From an experimental perspective, this method requires two key hyperparameters and needs different hyperparameters for different datasets, which is a disadvantage compared to other methods.

Author response:
>From an experimental perspective, this method requires two key hyperparameters and needs different hyperparameters for different datasets, which is a disadvantage compared to other methods.

As we discussed in our limitation section, our method do need two hyperparameters, however, we find in practice that the tuning of $\eta$ is not sensitive.
To validate this claim, we conduct ablation studies on $\eta$ using all D4RL datasets mentioned in Section 5.1. Experimental details remain consistent with those outlined in Appendix D, and the results are presented below. For MuJoCo datasets, a higher value of $\eta$ results in little or no performance drop, while a lower value of $\eta$ may induce instability and lead to small decrease in performance. In the case of AntMaze datasets, O-DICE also exhibits robustness to variations in $\eta$. O-DICE consistently outperforms the majority of offline RL methods across a wide range of $\eta$ values.

|                 | $\eta=0.2$ | $\eta=0.4$ | $\eta=0.6$ | $\eta=0.8$ | $\eta=1.0$ |
| --------------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| hopper-m        | 75.1       | 80.3       | 84.3       | 84.1       | 86.1       |
| hopper-m-r      | 94.4       | 99.1       | 98.3       | 98.6       | 99.9       |
| hopper-m-e      | 108.6      | 109.3      | 110.8      | 109.9      | 110.8      |
| walker-m        | 84.9       | 84.2       | 83.2       | 79.5       | 79.9       |
| walker-m-r      | 62.8       | 74.2       | 83.6       | 79.9       | 77.1       |
| walker-m-e      | 110.8      | 110.9      | 110.2      | 110.7      | 110.9      |
| halfcheetah-m   | 47.4       | 47.2       | 46.6       | 46.8       | 46.9       |
| halfcheetah-m-r | 44.0       | 43.6       | 42.4       | 43.0       | 42.9       |
| halfcheetah-m-e | 93.2       | 92.8       | 91.1       | 92.1       | 91.2       |
| antmaze-u       | 92.0       | 90.6       | 92.4       | 92.3       | 94.1       |
| antmaze-u-d     | 68.7       | 70.3       | 72.0       | 74.0       | 79.5       |
| antmaze-m-p     | 86.0       | 78.3       | 70.0       | 61.9       | 55.1       |
| antmaze-m-d     | 82.7       | 75.0       | 68.3       | 57.5       | 49.9       |
| antmaze-l-p     | 36.1       | 44.0       | 52.7       | 55.9       | 32.0       |
| antmaze-l-d     | 38.7       | 46.3       | 52.0       | 54.0       | 30.5       |

Reviewer comment (Theory):
The author claims that orthogonal-gradient can help consider state-level constraints, but there is no explicit explanation in the analysis as to why this constraint exists. Please provide a detailed explanation.

Author response:
>The author claims that orthogonal-gradient can help consider state-level constraints, but there is no explicit explanation in the analysis as to why this constraint exists. Please provide a detailed explanation.

Theorem 4 is not trying to prove that $\epsilon$ will always exist. Rather, we aim to demonstrate that, with a smaller value of $\Psi_{\theta}(s, s')$, the inequality condition $\Psi_{\theta}(s, s')>M-C\cdot \| \epsilon \|^2$ requires $\epsilon$ to have a larger norm in order to change the sign of $V(s'+\epsilon) - V(s)$. In other words, the existence of $\epsilon$ capable of causing a sign flip in $V(s'+\epsilon) - V(s)$ must satisfy the condition $\Psi_{\theta}(s, s')>M-C\cdot \| \epsilon \|^2$. This condition can be reformulated as $\| \epsilon \|^2 > \frac{M - \Psi_{\theta}(s, s')}{C}$, indicating that it becomes harder for $V(s'+\epsilon) - V(s)$ to flip sign with a smaller $\Psi_{\theta}(s, s')$. 

If the sign (or value) of $V (s') − V (s)$ can be easily flipped by some noise $\epsilon$, first the policy extraction part during training will be affected (as two similar s′ will have totally different BC weight), then the optimal policy induced from V is likely to be misled by OOD states encountered during evaluation and take wrong actions. 
For example, for transition $<s,a_1, s'_1>$ in the offline dataset, suppose $s'_1$ is more near the boundary of dataset support than $s$ and the true value of $V(s'_1)$ is smaller than $V(s)$, suppose another transition $<s,a_2, s'_1+\epsilon>$ is also in the dataset. Then the policy will be misled by $a_2$ because the learned $V$ will be affected by the small noise $\epsilon$, according to Theorem 4. If it takes $a_2$ then it will go to OOD states more easily. An ideal choice is do not believe either $a_1$ and $a_2$ because they are equally bad, by using orthogonal gradient update we can better do that.

Reviewer comment (Theory):
Is the $\epsilon$ in Theorem 4 different for different $s$? If so, I think Theorem 4 is trivial since we can easily find a state-wise $epsilon(s)$ to make $V(s’+\epsilon)-V(s)$ flip sign.

Author response:
>I think Theorem 4 is trivial since we can easily find an state-wise $\epsilon$ to flip sign.

Theorem 4 is not trying to prove that $\epsilon$ will always exist. Rather, we aim to demonstrate that, with a smaller value of $\Psi_{\theta}(s, s')$, the inequality condition $\Psi_{\theta}(s, s')>M-C\cdot \| \epsilon \|^2$ requires $\epsilon$ to have a larger norm in order to change the sign of $V(s'+\epsilon) - V(s)$. In other words, the existence of $\epsilon$ capable of causing a sign flip in $V(s'+\epsilon) - V(s)$ must satisfy the condition $\Psi_{\theta}(s, s')>M-C\cdot \| \epsilon \|^2$. This condition can be reformulated as $\| \epsilon \|^2 > \frac{M - \Psi_{\theta}(s, s')}{C}$, indicating that it becomes harder for $V(s'+\epsilon) - V(s)$ to flip sign with a smaller $\Psi_{\theta}(s, s')$. 

If the sign (or value) of $V (s') − V (s)$ can be easily flipped by some noise $\epsilon$, first the policy extraction part during training will be affected (as two similar s′ will have totally different BC weight), then the optimal policy induced from V is likely to be misled by OOD states encountered during evaluation and take wrong actions. 
For example, for transition $<s,a_1, s'_1>$ in the offline dataset, suppose $s'_1$ is more near the boundary of dataset support than $s$ and the true value of $V(s'_1)$ is smaller than $V(s)$, suppose another transition $<s,a_2, s'_1+\epsilon>$ is also in the dataset. Then the policy will be misled by $a_2$ because the learned $V$ will be affected by the small noise $\epsilon$, according to Theorem 4. If it takes $a_2$ then it will go to OOD states more easily. An ideal choice is do not believe either $a_1$ and $a_2$ because they are equally bad, by using orthogonal gradient update we can better do that.

Reviewer comment (Theory):
The algorithmic complexity of O-DICE has not been analyzed.

Author response:
>The algorithmic complexity of O-DICE has not been analyzed.

As evident in the pseudocode, O-DICE only needs to exclusively compute the backward gradient and perform gradient projection. The algorithmic complexity of these two steps is equivalent to calculating the forward gradient, a step common to all other DICE algorithms. Consequently, O-DICE's algorithmic complexity is no heavier than that of other DICE algorithms.