Paper: Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
Training world models might be very time-consuming.

Author response:
## Training time for world model and for algorithms with and without COCOA
Learning dynamics is not a significant burden compared to policy training. The time taken to train dynamics models for each algorithm is about 5-10 percent of the policy training time.
For IQL as an example, it takes about 1-2 hours on training dynamics, just a few minutes for anchor seeker training and 13 hours for policy training on TITAN RTX GPU. The training time per epoch increased from 11 seconds to 24 seconds as of now.

We mainly focused on the performance improvement that COCOA can bring to existing offline RL algorithms rather than optimizing the running time of our implementation. The majority of the additional training time by COCOA involves frequent calls to the pretrained anchor seeker policy, the dynamics model and the main policy. We expect that training time with COCOA can be greatly reduced through the operator fusion using a toolbox like TorchInductor.

Reviewer comment (Evaluation):
Proposed approach lead to the improvement only in approximately 58% of cases. And performance might be decreased dramatically. I've also calculated the differences between "+ COCOA" and "Alone" in Table 1 and it appeared that COCOA decreases performance down by 0.8 points. This indicates that only specific algorithms are expected to benefit on average by COCOA.

Author response:
## Overall performance with COCOA
Upon investigation following the reviewer's suggestion, we present our updated experimental results with the layer normalization for the anchor seeking policy and the transductive predictor, which turned out to be the missing component for stable training of COCOA. The results are shown in Table 1 in the paper. Comparison with reproduced IQL performance on Gym-MuJoCo random tasks are also included in Appendix. Note that we add the layer normalization only to the COCOA-specific components for a fair comparison with the baselines. As a result, COCOA increases the average performance of all base algorithms evaluated on the D4RL Gym-MuJoCo benchmark. The winning task ratio also becomes 63%. This suggests that with a proper training stabilization, adding COCOA can be beneficial across diverse base algorithms.

Reviewer comment (Experiments):
For MOPO and MOBILE hyperparameters were heavily tuned for each dataset which might be the cause why they benefited.

Author response:
## Hyperparameter sensitivity
Regarding MOPO and MOBILE's hyperparameter sensitivity, due to the time and resource constraints, we could not perform exhaustive analysis like EOP. We found that the `rollout length` and `penalty coefficient` need to be slightly larger than those used in the existing MOPO and MOBILE methods. While MOPO and MOBILE conduct the searches for `rollout length` and `penalty coefficient` with 2 and 4, or 2 and 5 values respectively, we use our search with 3 and 5 values.

Reviewer comment (Experiments):
Evaluation is performed only on the Gym MuJoCo datasets which I think is not enough now and evaluation on D4RL AntMaze or Adroit is essential.

Author response:
## Other benchmarks for COCOA
Following the reviewer’s suggestion, we experiment with two more benchmarks - D4RL Adroit and NeoRL [5]. The outcomes of these experiments are summarized in Table 2, 3 in our response to Reviewer HX7o.
This broader analysis reveals that COCOA significantly enhances the performance of IQL and MOBILE in most tasks. For a detailed insight into the experimental setup, please refer to the updated paper. 

Our method consistently enhances the performance across six tested D4RL Adroit tasks, showcasing its robustness and adaptability. In the difficult tasks like the door and hammer due to sparse rewards, the baselines work poorly, and our COCOA can still improve them. On the other hand, in the pen task, our method achieves significant performance improvements.




We also verify the effectiveness of our algorithm in the NeoRL benchmark, known for its conservative, less exploratory data generation compared to the D4RL Gym-MuJoCo. Although its state-action space coverage is highly limited, our preliminary experiments, conducted using MOPO, show significant performance improvement, suggesting potential benefits for other algorithms as well, which will be done for the camera-ready. 

Due to the time constraint of the rebuttal period, we test COCOA on six D4RL Adroit tasks on top of IQL and MOBILE, with plans to expand to CQL, MOPO, and ReBRAC in the camera-ready, and initially tested only with MOPO for NeoRL, intending to examine more algorithms later.

Reviewer comment (Evaluation):
What is the training time for algorithms with and without COCOA?

Author response:
## Training time for world model and for algorithms with and without COCOA
Learning dynamics is not a significant burden compared to policy training. The time taken to train dynamics models for each algorithm is about 5-10 percent of the policy training time.
For IQL as an example, it takes about 1-2 hours on training dynamics, just a few minutes for anchor seeker training and 13 hours for policy training on TITAN RTX GPU. The training time per epoch increased from 11 seconds to 24 seconds as of now.

We mainly focused on the performance improvement that COCOA can bring to existing offline RL algorithms rather than optimizing the running time of our implementation. The majority of the additional training time by COCOA involves frequent calls to the pretrained anchor seeker policy, the dynamics model and the main policy. We expect that training time with COCOA can be greatly reduced through the operator fusion using a toolbox like TorchInductor.

Reviewer comment (Experiments):
What is hyperparameters sensitivity for MOPO and MOBILE when COCOA is added compared to the same algorithms without COCOA? EOP (https://arxiv.org/abs/2110.04156) can be used for this purpose.

Author response:
## Hyperparameter sensitivity
Regarding MOPO and MOBILE's hyperparameter sensitivity, due to the time and resource constraints, we could not perform exhaustive analysis like EOP. We found that the `rollout length` and `penalty coefficient` need to be slightly larger than those used in the existing MOPO and MOBILE methods. While MOPO and MOBILE conduct the searches for `rollout length` and `penalty coefficient` with 2 and 4, or 2 and 5 values respectively, we use our search with 3 and 5 values.

Reviewer comment (Writing):
Minor text issues: the second paragraph of section 2.1 duplicates information from the first one. Results section has "**FFor** IQL,".

Author response:
We appreciate the reviewer’s thoughtful and constructive feedback. We revised multiple text issues that the reviewer pointed out. Due to space limits, tables were deferred to our response to Reviewer HX7o or can be found in the paper.

Reviewer comment (Experiments):
Why not train a reverse policy but use a random divergent reverse policy? Is there an ablation study on this?

Author response:
An effective reverse rollout policy should diverge from an offline dataset and encompass various terminal states. We avoided training a reverse policy because defining a precise loss function for it would involve quantifying (1) the state's distance from the dataset distribution and (2) the terminal state diversity, which is computationally intensive.

Hence, we employed a random divergent reverse policy while comparing COCOA with trained reverse policy as an ablation study. Similar to [1], we employed a state-conditioned conditional variational autoencoder[2] that mimics the offline dataset's behavior in reverse. The performance of MOPO+COCOA on two D4RL Gym-MuJoCo tasks showed that a random diverse policy is superior for high performance.

Table 1:  Ablation study on the reverse policy. 
| Task | Trained | Random |
|----------------------------|------------|----------|
| halfcheetah-medium-replay | 58.8 | 72.1 |
| walker2d-medium-replay | 68.6 | 92.3 |

Reviewer comment (Evaluation):
The evaluation is only limited to D4rl benchmarks and why in IQL, adding COCOA does not improve much among all the d4rl tasks, could you provide some insights into this?

Author response:
Following the reviewer’s suggestion, we experimented with two more benchmarks - D4RL Adroit and NeoRL [3]. The outcomes of these experiments are summarized in the tables below.
This broader analysis reveals that COCOA enhances the performance of IQL and MOBILE in most tasks. For a detail about experimental setup, refer to the updated paper. 

Our method demonstrated consistent performance enhancements across six D4RL Adroit tasks we tested, showcasing its robustness and adaptability. While COCOA encountered challenges in complex tasks like the door and hammer, akin to its original algorithm, this reflects the inherent difficulty of these tasks due to sparse rewards. Notably, in tasks like the pen, our method achieved noticeable performance improvements.

We also verified the effectiveness of our algorithm in the NeoRL benchmark, which is known to be more difficult than the traditional D4RL Gym-MuJoCo benchmark as it generates the data in a more conservative and less exploratory manner. The data collected in this manner have less coverage in the state-action space, posing the need for more fine-grained stitching. In this perspective, we can see that the dataset's challenges align with our algorithm.
To check whether our algorithm also performs well in NeoRL, we firstly conducted experiments with MOPO which demonstrates an impressive performance improvement. Based on these enhancements, we anticipate that other algorithms would also have a big advantage with our algorithm.
Under time constraints, we tested COCOA on six D4RL Adroit tasks on top of IQL and MOBILE, with plans to expand to CQL, MOPO, and ReBRAC in the final paper, and initially tested only with MOPO for NeoRL, intending to examine more algorithms later.

Table 2: D4RL Adroit benchmark results.
| Task | IQL | IQL+COCOA | MOBILE | MOBILE+COCOA |
|------------------|--------|-----------|---------|--------------|
| door-cloned-v1 | 2.11 | 2.81 | -0.27 | 0.92 |
| door-human-v1 | 5.27 | 6.81 | -0.28 | -0.05 |
| hammer-cloned-v1 | 0.53 | 0.57 | 0.23 | 0.29 |
| hammer-human-v1 | 1.33 | 2.21 | 0.25 | 1.11 |
| pen-cloned-v1 | 72.55 | 73.98 | 54.95 | 48.9 |
| pen-human-v1 | 74.4 | 78.15 | 14.76 | 41.65 |
| **Average** | **26.03** | **27.22** | **11.61** | **15.47** |


Table 3: NeoRL benchmark results.
| Task | MOPO | MOPO+COCOA |
|-----------------|-------|------------|
| HalfCheetah-L | 40.1 | 47.47 |
| Hopper-L | 6.2 | 23.02 |
| Walker2d-L | 11.6 | 14.23 |
| HalfCheetah-M | 62.3 | 78.26 |
| Hopper-M | 1 | 61.65 |
| Walker2d-M | 39.9 | 49.80 |
| HalfCheetah-H | 65.9 | 42.44 |
| Hopper-H | 11.5 | 29.31 |
| Walker2d-H | 18 | 53.38 |
| **Average** | **28.5** | **44.40** |

## The performance of IQL

Especially in the context of IQL, we observed that certain tasks occasionally exhibited a sudden drop in performance. Based on the guidance you provided, and considering that model-free algorithms are typically more challenging to train compared to model-based algorithms (as they do not augment the dataset with a dynamics model), we finalized our network design to employ the layer normalization technique. This approach is particularly much more effective in tasks especially such as 'hopper-medium-replay', 'walker 2d-medium-replay' and so forth.