Paper: HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approach (ICLR-2024)

Reviewer comment (Experiments):
In Section 3, the paper compares the results of the downstream offline RL algorithm using two types of augmented data: high-diversity data and high-return data. The high-return data was generated from a well-trained offline policy with a higher return. Since it is well-known that offline RL algorithms tend to perform better on medium-expert datasets than medium datasets, I believe it is evident that the downstream offline RL algorithm performs better with high-return augmented data. The two augmented data should have a similar maximum return value for a fair comparison.

Author response:
In section 3, we would like to explore whether augmenting high-reward (but low-diversity) data or high-diversity (low-reward) data is more effective. We agree that it is evident that the downstream offline RL algorithm performs better with high-return augmented data. However,  in offline setting, augmentation for a specific dataset does not allow generating data that is too far from the dataset, e.g., it is not possible to generate expert data when only the medium dataset is known. Intuitively, diverse data can expand the dataset coverage and thus improve the downstream algorithm generalization, which may also be beneficial to the dataset. So we think it makes sense to investigate which of these two types of data is better in data augmentation for offline RL. In fact, in the problem of data augmentation in offline RL for specific datasets, there is no previous work that explores exactly what characteristics of data are better to use.

In summary, section 3 provides a fair comparison of the two data types and suggests that generating high-return data for a specific dataset may improve offline RL more.

Reviewer comment (Novelty):
The paper compares the performance of HIPODE with CABI. HIPODE and CABI share common aspects, such as using forward and inverse dynamics models. A careful analysis of how they differ from each other would be helpful for the readers to understand the novelty of HIPODE.

Author response:
1. CABI does not consider return, but HIPODE focuses on generating high-return data.

2. CABI use a bi-direction generation, but HIPODE only generates transition in the forward direction.

3. CABI generates s' by (s,a), in both forward or reverse rolling. HIPODE generates (s,s') and then uses a reverse dynamic model to generate action.

4. HIPODE uses negative sampling to avoid OOD state to ensure data authenticity, while CABI uses double check mechanism.

Reviewer comment (Experiments):
My second concern is that the method seems to focus on generating high-value augmented data with low diversity (or low state-action coverage), but as mentioned above, data diversity is also quite important to the success of data augmentation. Thus, HIPODE may have limited applicability. I don't consider this a huge drawback though; in principle, one could use HIPODE to generate high-value augmented data along with an augmentation strategy that generates highly diverse (and potentially low-value) augmented data. It would be interesting to run experiments with "Original + Return + Diversity X" augmented data (using the naming convention of Table 1).

Author response:
## What about augmenting with both high-quality and high-diversity data?

This is really a valuable idea. We have obtained results on TD3BC and IQL about this. The results are as follows.

|                 | dataset          | TD3BC        | IQL          |
| --------------- | ---------------- | ------------ | ------------ |
| halfcheetah-r   | Org              | 12.8$\pm$1.9 | 16.5$\pm$0.6 |
|                 | Org+Qua          | 25.8$\pm$0.0 | 21.2$\pm$4.7 |
|                 | Org+Qua+Div 0.01 | 25.9$\pm$0.3 | 19.8$\pm$3.1 |
|                 | Org+Qua+Div 0.1  | 25.8$\pm$0.2 | 19.9$\pm$3.9 |
|                 | Org+Qua+Div 1.0  | 25.5$\pm$0.7 | 25.0$\pm$0.3 |
| halfcheetah-m-r | Org              | 43.3$\pm$0.6 | 41.1$\pm$1.1 |
|                 | Org+Qua          | 46.8$\pm$0.1 | 44.6$\pm$0.2 |
|                 | Org+Qua+Div 0.01 | 46.5$\pm$0.1 | 44.8$\pm$0.1 |
|                 | Org+Qua+Div 0.1  | 46.7$\pm$0.2 | 44.9$\pm$0.1 |
|                 | Org+Qua+Div 1.0  | 45.3$\pm$0.5 | 43.6$\pm$0.1 |

In the present experiment, the quantities of high-quality and diversity data were both half of the original data. The experimental results show that adding only high quality data significantly improves the performance compared to the original data. And then after continuing to add diversity data, the performance is almost unchanged.  This suggests that high quality data plays a greater role in data augmentation to improve the offline RL performance, when all data is completely realistic .

Reviewer comment (Experiments):
I'm skeptical about using learned model to generate rewards. Offline RL requires access to a reward function, so why not simply label augmented transitions with their true reward? Generating augmented rewards seems like an unnecessary source of variability.

Author response:
## Offline RL requires access to a reward function. 

Our offline problem setting is only knowing the dataset and nothing else. The reward function can be seen as part of the environment which is not accessible. to the best of our knowledge, offline RL algorithms such as CQL or TD3BC are also set up in this way.

Reviewer comment (Evaluation):
The paper should include significance test for results reported in Table 1 and Table 2 (e.g. paired-t test at a 95% confidence level). For instance, in Table 1, the “Original + Return” returns for “halfcheetah-m-r” look very similar to returns for the other datasets.

Author response:
## Significance test for results 

We have counted the standard deviations of both Tabel1, Table 2 and Table 3 in the paper as follows.

Table 1:

| task name       | augmenting type         | td3bc        | cql           | iql          |
| --------------- | ----------------------- | ------------ | ------------- | ------------ |
| halfcheetah-r   | original                | 12.8$\pm$1.9 | 17.0$\pm$9.3  | 16.5$\pm$0.6 |
|                 | original+diversity 0.01 | 12.1$\pm$0.6 | 2.0$\pm$0.2   | 11.2$\pm$4.0 |
|                 | original+diversity 0.1  | 12.0$\pm$0.5 | 16.1$\pm$5.5  | 14.3$\pm$3.0 |
|                 | original+diversity 1.0  | 9.2$\pm$0.7  | 3.0$\pm$0.4   | 12.6$\pm$6.2 |
|                 | original+return         | 25.8$\pm$0.0 | 23.8$\pm$0.3  | 21.2$\pm$4.7 |
| halfcheetah-m-r | original                | 43.3$\pm$0.6 | 42.5$\pm$0.9  | 41.1$\pm$1.1 |
|                 | original+diversity 0.01 | 44.6$\pm$0.6 | 38.4$\pm$6.7  | 43.3$\pm$0.3 |
|                 | original+diversity 0.1  | 44.3$\pm$0.3 | 1.8$\pm$0.1   | 44.0$\pm$0.2 |
|                 | original+diversity 1.0  | 41.8$\pm$0.7 | 26.6$\pm$19.2 | 40.1$\pm$0.3 |
|                 | original+return         | 46.8$\pm$0.1 | 52.6$\pm$0.1  | 44.6$\pm$0.2 |

Score of 12 Mujoco-v0 datasets of Tabel 2:

| Task Name       | CQL+ HIPODE     | CQL+ CABI        | CQL           | TD3BC+ HIPODE    | TD3BC+ CABI    | TD3BC         |
| --------------- | --------------- | ---------------- | ------------- | ---------------- | -------------- | ------------- |
| halfcheetah-m-e | $99.5\pm{4.5}$  | $101.0$ $\pm$1.2 | 94.8$\pm$3.9  | $102.6\pm{2.2}$  | 94.6$\pm$8.5   | 98.0$\pm$3.1  |
| hopper-m-e      | $112.1\pm{0.1}$ | 112.0 $\pm$0.1   | 111.9$\pm$0.1 | $112.4\pm{0.3}$  | 102.8$\pm$16.0 | 112.0$\pm$0.1 |
| walker2d-m-e    | $94.0\pm{4.0}$  | 92.3$\pm$13.8    | 70.3$\pm$5.7  | $105.3\pm{4.0} $ | 99.6$\pm$6.4   | 105.4$\pm$3.9 |
| halfcheetah-m-r | $46.0\pm{0.1}$  | 42.8$\pm$2.0     | 42.5$\pm$0.9  | $44.0\pm$0.7 $  | 43.4$\pm$0.5   | 43.3$\pm$0.6  |
| hopper-m-r      | $34.7\pm{2.9}$  | 29.7$\pm$2.7     | 28.2$\pm$1.2  | $36.2\pm{0.8} $  | 32.7$\pm$2.3   | 32.7$\pm$0.5  |
| walker2d-m-r    | $21.7\pm{6.5}$  | 12.7$\pm$6.7     | 5.1$\pm$3.6   | $36.4\pm{11.1}$  | 37.7$\pm$15.4  | 19.6$\pm$8.6  |
| halfcheetah-m   | $39.3\pm{0.2}$  | 36.7$\pm$0.2     | 39.2$\pm$0.4  | $43.7\pm$0.5$   | 43.0$\pm$0.3   | 43.7$\pm$0.4  |
| hopper-m        | $30.4\pm{0.9}$  | 30.5$\pm$0.3     | 30.3$\pm$0.7  | $99.9\pm$0.3$  | 99.7$\pm$0.2   | 99.9$\pm$0.9  |
| walker2d-m      | $75.7\pm{5.1}$  | 46.8$\pm$16.1    | 66.9$\pm$9.3  | $80.1\pm$0.7$  | 80.3$\pm$2.0   | 79.7$\pm$1.8  |
| halfcheetah-r   | $23.1\pm{1.5}$  | 2.8$\pm$0.9      | 17.0$\pm$5.7  | $15.5\pm$0.9$  | 12.5$\pm$1.5   | 12.8$\pm$1.9  |
| hopper-r        | $10.4\pm{0.1}$  | 10.2$\pm$0.1     | 10.4$\pm$0.1  | $10.9\pm$0.2 $  | 10.9$\pm$0.0   | 10.9$\pm$0.2  |
| walker2d-r      | $10.5\pm{0.6}$  | -0.1$\pm$0.0     | 1.7$\pm$0.9   | $6.6\pm$1.0$  | 3.0$\pm$0.7    | 0.37$\pm$0.1  |
| Average         | 49.8            | 43.1             | 43.2          | 57.8             | 55.0           | 54.9          |

Table 3:

| Task Name       | Repeat+TD3BC  | NoV+TD3BC     | HIPODE+TD3BC  | TD3BC         |
| --------------- | ------------- | ------------- | ------------- | ------------- |
| halfcheetah-m-e | 97.4$\pm$3.6  | 99.1$\pm$4.2  | 102.6$\pm$2.2 | 98.0$\pm$3.7  |
| hopper-m-e      | 111.9$\pm$0.5 | 110.5$\pm$3.5 | 112.4$\pm$0.3 | 112$\pm$0.1   |
| walker2d-m-e    | 38.0$\pm$64.4 | 102.5$\pm$5.6 | 105.3$\pm$4.0 | 105.4$\pm$3.9 |
| halfcheetah-m-r | 42.5$\pm$1.6  | 44.0$\pm$0.2  | 44.0$\pm$0.7  | 43.3$\pm$0.6  |
| hopper-m-r      | 36.5$\pm$4.2  | 36.2$\pm$8.0  | 36.2$\pm$0.8  | 32.7$\pm$0.9  |
| walker2d-m-r    | 18.0$\pm$14.4 | 30.0$\pm$4.1  | 36.4$\pm$11.1 | 19.6$\pm$8.6  |
| halfcheetah-m   | 43.6$\pm$0.8  | 43.1$\pm$0.5  | 43.7$\pm$0.5  | 43.7$\pm$0.4  |
| hopper-m        | 99.8$\pm$0.1  | 99.7$\pm$0.3  | 99.9$\pm$0.3  | 99.9$\pm$0.5  |
| walker2d-m      | 79.5$\pm$2.7  | 79.7$\pm$3.1  | 80.1$\pm$0.7  | 79.7$\pm$1.8  |
| halfcheetah-r   | 11.7$\pm$0.3  | 13.1$\pm$1.4  | 15.5$\pm$0.9  | 12.8$\pm$1.9  |
| hopper-r        | 11.1$\pm$0.0  | 10.9$\pm$0.1  | 10.9$\pm$0.2  | 10.9$\pm$0.2  |
| walker2d-r      | 1.9$\pm$0.4   | 2.1$\pm$0.7   | 6.6$\pm$1.0   | 0.4$\pm$0.1   |
| halfcheetah-e   | 104.2$\pm$3.3 | 105.3$\pm$0.7 | 106.5$\pm$0.5 | 107.0$\pm$0.9 |
| hopper-e        | 112.5$\pm$0.5 | 112.3$\pm$0.2 | 112.3$\pm$0.5 | 107.3$\pm$8.7 |
| walker2d-e      | 78.1$\pm$48.5 | 104.9$\pm$5.8 | 106.6$\pm$5.2 | 98.7$\pm$6.0  |
| Average         | 59.1$\pm$9.7  | 66.2$\pm$2.6  | 67.9$\pm$1.9  | 64.7$\pm$2.6  |

Reviewer comment (Experiments):
In the experiments, the authors only test the proposed approach in environments where the agent’s action has a cyclic pattern. For these tasks, the transition model could be relatively simple and easy to be learned.

Author response:
We realized this as well. We also evaluate HIPODE+TD3BC on the -human datasets of Adroit, a  robotic manipulation environment and find that HIPODE also possesses gain effects, as shown in Table 9 in the Appendix.

Reviewer comment (Experiments):
The experiments with only 3 random seeds are unreliable in terms of reinforcement learning.

Author response:
We evaluate HIPODE+IQL on the Mujoco-v2 datasets on 4 random seeds as shown in Table 8 in Appendix  C.2. We are also adding seed to the main experiment in Section 5.1 and will update the results.

Reviewer comment (Theory):
It seems that there is no theoretical statement for the proposed approach.

Author response:
We recognize that we are currently deficient in theory, and that some theoretical issues have yet to be proven. For example, Can high quality data augmentation really leads to a theoretical gain? We will continue to think about this in the future.

Reviewer comment (Experiments):
Is the proposed approach applicable to the environment with discrete action space? It would be great if the authors could provide more experiments on different environments, such as Atari.

Author response:
We will evaluate HIPODE on Atari. However, the Atari experiment is considered to be a bit challenging because the discrete actions can cause the generation of exactly the same data as the dataset. A recent work [1] tries data augmentation on Atari but finds that data augmentation can have negative impacts on downstream policy learning algorithms.

[1] Hwang D, Park M, Lee J. Sample Generations for Reinforcement Learning via Diffusion Models[J]. 2023.

Reviewer comment (Evaluation):
It is interesting that the proposed method significantly outperforms previous methods on walker. What’s the cause of it?

Author response:
We have also observed this interesting result. A recent work [2] finds that a model learnt on an offline dataset can plan a precise trajectory with explicit OOD actions on the Walker2d environment. Thus, we believe that a dynamics model with a strong ability to generalize to novel synthetic data is easier to learn on Walker2d.

HIPODE generates high-quality data that leads to substantial improvements across nearly all datasets. The varying degrees of improvement on different datasets may be attributed to factors such as the environment, dataset characteristics, and variations in downstream algorithms employed. 

[2]  Dong Z, Yuan Y, Hao J, et al. AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model[J]. arXiv preprint arXiv:2310.02054, 2023.

Reviewer comment (Experiments):
Relevance of Dataset Quality Analysis: Section 3's analysis, while potentially informative, may have limited relevance to the paper's contributions. The use of the true environment for generating synthetic data is an idealized condition not typically accessible in practical Offline RL scenarios where learned dynamics models are employed. This idealization might overstate the effectiveness of high-return data augmentation as it does not account for inaccuracies that would be present when using a learned model for data generation. Moreover, the assertion that high-value states improve policy learning is somewhat tautological and may not be particularly insightful since it is well-understood that higher-quality datasets tend to yield better-performing policies.

Author response:
We agree that the true environment is an idealized condition. What we want to explore in this section is whether it is more beneficial to augment high-return data or high diversity data. In order to do this, we need to control the authority of both types of data to be the same, so we consider making both types of augmentation data completely true to avoid the authority affecting the performance. We considered that the ground truth data cannot be used in the offline setting, so we set the augmentation strategy to an offline rl strategy, avoiding generating ood action, which is similar to the data augmentation that works in the offline setting.

We also agree that high-value states improve policy learning is somewhat tautological. However, we believe that there may be the influence of other factors in the offline rl data. For example, intuitively, Diverse data expands the dataset coverage and thus improves generalization of the downstream policy, e.g., [1] concluded that noise data is beneficial for OFFLINE RL, but there setup is different from data augmentation for a specific dataset. Data augmentation for a specific dataset in the OFFLINE setting has some unique characteristics, e.g., Too large proportion of synthetic data and too much noise can be detrimental to policy learning. In fact, no previous work has explored what's better for data augmentation to specific datasets in offline setting.

As a summarize, section3 compares the two kinds of data as fairly as possible while staying close to the data augmentation setting, and finds out that it is better to do data augmentation for a specific dataset with high-return data.

Reviewer comment (Evaluation):
Strength of Empirical Results: The empirical results presented, particularly in Section 5.2, lack sufficient detail and statistical rigor to substantiate strong claims of improvement. The absence of confidence intervals or a deeper statistical analysis in Table 3 makes it difficult to discern the significance of the performance gains attributed to HIPODE. Without this, the evidence provided does not firmly establish HIPODE’s superiority over, e.g., the NoV method.

Author response:
We report the standard deviations in Table 3.

| Task Name       | Repeat+TD3BC  | NoV+TD3BC     | HIPODE+TD3BC  | TD3BC         |
| --------------- | ------------- | ------------- | ------------- | ------------- |
| halfcheetah-m-e | 97.4$\pm$3.6  | 99.1$\pm$4.2  | 102.6$\pm$2.2 | 98.0$\pm$3.7  |
| hopper-m-e      | 111.9$\pm$0.5 | 110.5$\pm$3.5 | 112.4$\pm$0.3 | 112$\pm$0.1   |
| walker2d-m-e    | 38.0$\pm$64.4 | 102.5$\pm$5.6 | 105.3$\pm$4.0 | 105.4$\pm$3.9 |
| halfcheetah-m-r | 42.5$\pm$1.6  | 44.0$\pm$0.2  | 44.0$\pm$0.7  | 43.3$\pm$0.6  |
| hopper-m-r      | 36.5$\pm$4.2  | 36.2$\pm$8.0  | 36.2$\pm$0.8  | 32.7$\pm$0.9  |
| walker2d-m-r    | 18.0$\pm$14.4 | 30.0$\pm$4.1  | 36.4$\pm$11.1 | 19.6$\pm$8.6  |
| halfcheetah-m   | 43.6$\pm$0.8  | 43.1$\pm$0.5  | 43.7$\pm$0.5  | 43.7$\pm$0.4  |
| hopper-m        | 99.8$\pm$0.1  | 99.7$\pm$0.3  | 99.9$\pm$0.3  | 99.9$\pm$0.5  |
| walker2d-m      | 79.5$\pm$2.7  | 79.7$\pm$3.1  | 80.1$\pm$0.7  | 79.7$\pm$1.8  |
| halfcheetah-r   | 11.7$\pm$0.3  | 13.1$\pm$1.4  | 15.5$\pm$0.9  | 12.8$\pm$1.9  |
| hopper-r        | 11.1$\pm$0.0  | 10.9$\pm$0.1  | 10.9$\pm$0.2  | 10.9$\pm$0.2  |
| walker2d-r      | 1.9$\pm$0.4   | 2.1$\pm$0.7   | 6.6$\pm$1.0   | 0.4$\pm$0.1   |
| halfcheetah-e   | 104.2$\pm$3.3 | 105.3$\pm$0.7 | 106.5$\pm$0.5 | 107.0$\pm$0.9 |
| hopper-e        | 112.5$\pm$0.5 | 112.3$\pm$0.2 | 112.3$\pm$0.5 | 107.3$\pm$8.7 |
| walker2d-e      | 78.1$\pm$48.5 | 104.9$\pm$5.8 | 106.6$\pm$5.2 | 98.7$\pm$6.0  |
| average         | 59.1$\pm$9.7  | 66.2$\pm$2.6  | 67.9$\pm$1.9  | 64.7$\pm$2.6  |