Paper: Offline Imitation Learning without Auxiliary High-quality Behavior Data (ICLR-2024)

Reviewer comment (Experiments):
The reward design seems to be too strict. Currently, judging from section 3.2, the reward for a state-action pair is 1 if and only if the state-action pair belongs to the expert dataset. However, with continuous state/action space and low quality auxiliary dataset, it is very likely that none of the state-action pairs in \(D^O\) has a reward of 1 – and with perturbation in the continuous environment, it is even likely that every state appearing in \(D^O \cup D^E\) is unique and no state in the auxiliary dataset \(D^O\) leads to any reward. In such a case, the optimization relies on the smoothing effect of the critic network. It is suggested to add an ablation on replacing the current, identical reward with the discriminator used in prior work such as DWBC, ORIL, or DemoDICE.

Author response:
We obtained this reward design by maximizing the expert-state distribution: samples in the expert dataset have a reward of 1, and samples in the auxiliary dataset have a reward of 0. This design may be conservative because we mainly focus on the utilization of low-quality data, thus only utilizing transition information, which is independent of behavior quality. When the data contains some expert-similar trajectories, we can also replace the current identical reward with the discriminator used in prior work to make use of it.
Here we provide the corresponding additional experimental results. Specifically, we use DWBC, and OTIL to annotate the reward labels for offline auxiliary data, and then apply the BCDP method with optimized rewards. 

|   | Strict-BCDP | DWBC-BCDP | OTIL-BCDP |
| --- | --- | --- | --- |
| maze2d-sparse-umaze-v1 | 132.+/-22.0 | 128.+/-12.9 | **137.+/-6.59** |
| maze2d-sparse-medium-v1 | 137.+/-11.4 | 138.+/-19.0 | **139.+/-9.84** |
| maze2d-sparse-large-v1 | 124.+/-22.0 | 111.+/-14.8 | **137.+/-19.8** |
| hopper-medium | **98.7+/-5.81** | 88.3+/-8.85 | 59.4+/-0.97 |
| halfcheetah-medium | 18.4+/-10.9 | 15.6+/-11.1 | **40.5+/-3.79** |
| walker2d-medium | **98.0+/-1.94** | 97.2+/-9.88 | 96.1+/-4.79 |
| ant-medium | 59.7+/-16.2 | 14.1+/-13.8 | **94.8+/-4.07** |


The results of BCDP with optimized rewards demonstrate its potential when combined with previous reward annotation methods. Especially when the auxiliary data contains some relatively high-quality data, the performance of the BCDP framework can be further improved. It clearly supports the advantages of our proposal on scalability. We have added them to Appendix C.6 as additional supplementary material.

Reviewer comment (Reproducibility):
The hyperparameter is not specified, and some auxiliary plots are missing. Some important hyperparameters are missing, e.g., weight for maximizing Q in the objective, network architecture, learning rate, number of steps to update, frequency of updating with policy gradient from critic. Also, the paper does not include learning curves.

Author response:
We have provided the detailed hyper-parameters and the learning curves in Appendix C.4 and C.9 respectively.

Reviewer comment (Presentation):
There is no discussion on the limitation of the current work, and it is encouraged to include this in the paper.

Author response:
In this work, our main focus is to remove the traditional assumption of behavior quality in auxiliary data and leverage the transition information to assist in imitation learning. It is worth noting that our approach may be conservative, particularly when high-quality data is available in the dataset. However, as you suggested, we can further enhance the performance of BCDP by combining previous reward annotation methods. We acknowledge that our exploration in this area is currently limited, we will combine these two different approaches of utilizing auxiliary data in future research to achieve a better balance between performance and robustness. Moreover, this study currently only considers the model-free solution. Exploring how to integrate existing model-based techniques to guide the agent toward expert-observed states is also a promising direction that we will attempt in future work.

Reviewer comment (Writing):
Typo – in "background and formulation", "... which is much cheap to obtain" should be "... which is much **cheaper** to obtain".

Author response:
Thanks for your suggestions, we have revised them in the new version.

Reviewer comment (Writing):
Typo – in figure 2, "How to achieve a expert-observed state" should be "How to achieve **an** expert-observed state".

Author response:
Thanks for your suggestions, we have revised them in the new version.

Reviewer comment (Writing):
Typo – in the second line of section 4, "relatuve" should be "relative".

Author response:
Thanks for your suggestions, we have revised them in the new version.

Reviewer comment (Writing):
Notation issue – the notation \(\mathbb{I}\) is not formally introduced, which is a crucial factor to the reward design.

Author response:
Thanks for your suggestions, we have revised them in the new version.

Reviewer comment (Novelty):
**Practical Implementation:** While the work combines offline IL with DP, it leads to the well-established and pre-existing paradigm of offline RL. Practical implementation of BCDP presented in Section 3.2 is akin to applying offline RL on a different dataset split. As the authors note, BCDP is a special case of TD3+BC. However, its differences from TD3+BC or other offline RL algorithms remain unclear.

Author response:
**W2 and Q2: Comparison between BCDP and TD3+BC**  

In this work, we formalize the utilization of low-quality data in offline imitation learning as an **expert-state distribution maximization** problem and it can be instantiated using different dynamic programming methods. We chose TD3 because it is concise and stable. This does not mean that BCDP is a special case of TD3+BC. The difference arises from **two completely different purposes**. The behavioral cloning in the original TD3+BC is for conservatism, to keep the learned policy close to the offline policy and eliminate the estimation bias of TD3. Our goal with behavior cloning is to follow the imitation learning of expert data. Our TD3 (DP part) on the offline dataset ($ D^O \cup D^E $) is to enable the agent to transition from expert-unobserved states to expert-observed states. Compared to previous imitation learning methods, this provides a new direction for the safe utilization of low-quality data, that is, using their transition information to help imitation learning.  

**We actually compared it with the TD3+BC in all settings**. As we claimed in the raw paper: “UDS (Yu et al., 2022) labels all rewards from the unlabeled datasets with 0 and utilizes offline reinforcement learning algorithms to train the agent on the merged dataset. We have selected TD3+BC as our most similar offline RL algorithm, which allows it to be considered as an ablation study of our approach.” In all experiments, the UDS represents the raw TD3+BC in the offline imitation learning setting, as an ablation. In the offline imitation learning setting, where all data is unlabeled, applying offline RL algorithms on the union dataset is equivalent to UDS (Unlabeled Data Sharing), where expert data has reward label 1 and auxiliary data has reward label 0. As shown in the original paper's results and corresponding discussions, TD3+BC performs unsatisfactorily when the behavior quality of offline data is low.

Reviewer comment (Experiments):
**Choice of Baselines:** While the authors state that their approach is similar to TD3+BC and offline RL, BCDP is not compared to these methods. Authors claim that "they have selected TD3+BC as our most similar offline RL algorithm, which allows it to be considered as an ablation study for our approach". However, an ablation study of TD3+BC with BCDP is missing. It would be worthwhile to compare BCDP with TD3+BC or a conservative algorithm such as CQL. This would help evaluate the utility of BCDP with established offline RL methods.

Author response:
**W3 and Q3: Comparision with more offline RL methods**  

Thank you for your interest in comparing BCDP with more offline RL methods. We provide here a comparison with some popular offline RL methods, including CQL, IQL, and BEAR. Furthermore, we can also implement different (offline) RL methods in the BCDP framework, as an alternative to TD3. 
|  | UDS(Ablation) |  |  |  | BCDP (Ours) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TD3+BC | IQL | CQL | BEAR | TD3 | IQL | CQL | BEAR |
| sparse-umaze-v1 | 91.1 ± 22.9 | 49.0 ± 8.88 | 83.7 ± 86.3 | 21.9 ± 6.60 | **132. ± 22.0** | 111. ± 36.6  | 103.± 18.4 |  94.5 ± 47.8 |
| sparse-medium-v1 | 97.0 ± 20.0 | 90.4 ± 29.9 | 67.3 ± 63.2 |  -1.5 ± 5.04 | **137. ± 11.4**  | 125. ± 20.1 | 83.3 ± 38.2 | 87.2 ± 62.4 |
| sparse-large-v1 | 134. ± 26.0  | 100. ± 17.5  | 164. ± 37.5 |  0.67 ± 5.50  | 124. ± 22.0  | **206. ± 14.9**  | 105. ± 13.8 | 16.0 ± 16.1 |
| hopper-random-v2 | 1.15 ± 0.46 | 5.76 ± 1.99 |  32.0 ± 2.64 |  4.86 ± 2.47 | **73.2 ± 9.66**  | 46.5 ± 29.8 | 59.0 ± 9.15  | 4.86 ± 2.47 |
| halfcheetah-random-v2 | 4.62 ± 1.15 | 2.25 ± 0.00  |  6.67 ± 1.63 |  2.25 ± 0.00 | **18.8 ± 6.11** | 3.27 ± 2.60  |  1.82 ± 0.52 |  3.54 ± 2.31 |
| walker2d-random-v2 | -.11 ± 0.00 | 45.3 ± 48.6 | 7.41 ± 0.60 | 0.90 ± 0.01 | **105.± 2.08** | 88.1 ± 32.7 |  96.1 ± 9.33 | 104. ± 3.72 |
| ant-random-v2  | 30.4 ± 2.99 | **71.0 ± 0.76** | 39.8 ± 6.35 | 30.9 ± 0.02 | 54.1 ± 9.01 | 62.3 ± 12.9 |  47.0 ± 20.1 | 58.6 ± 4.30 |

From the results, we could find that our BCDP has scalability for different offline reinforcement learning methods. Compared to directly applying offline reinforcement learning methods (UDS), BCDP brings consistent performance improvement. These results further demonstrate the scalability of our BCDP. We have added these results to  Appendix C.7.

Reviewer comment (Presentation):
Table 1: the second-best result is not underlined.

Author response:
**Weakness: One minor issue: Table 1: the second-best result is not underlined.**  

We have revised it in the new version.

Reviewer comment (Evaluation):
From Table 1, we can observe that BCDP performs very well for random or low-quality data. But when more expert knowledge is included, BCDP has inferior results than others. Please explain why BCDP has less effective performance for non-random and human datasets.

Author response:
**Q2: Explanations on non-random and human datasets.**  

In this paper, we mainly focus on whether it is possible to remove previous assumptions on high-quality behavioral data and attempt to take advantage of the benefits of low-quality data. When the data contains some high-quality behavioral data, our method only uses the transition information in the data. This may be conservative, so its performance improvement is not as significant as in the setting with purely low-quality data. Fortunately, as suggested by reviewer uGnS, in this case, we can also combine previous research, such as DWBC or OTIL, to identify potential expert-similar data and further enhance performance. The corresponding experimental results are provided in Appendix C.6.    

In human data, the number of auxiliary data is limited (25 trajectories per task), which makes the transition information they can provide relatively limited. Therefore, performance improvement brought by BCDP is not very significant.