Paper: When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR-2024)

Reviewer comment (Evaluation):
Although explanations are provided for most of the observed facts, most of them are speculative without further evidence. For example, in Sec. 4.1, “a likely reason … mean CQL must propagate TD errors …”. If these conjectures are dived deeper with further evidence and justification, it will make the paper more solid.

Author response:
We would like to express our gratitude to the reviewer for providing us with thorough feedback.

1. We have addressed the explanation concern in Common Response section

Reviewer comment (Experiments):
The algorithms CQL and BC are no longer the SOTA results for these offline RL baselines, and more effective algorithms have been proposed with improvements. Examples include TD3BC [1], percentage BC, Diffusion Q-learning [2] for BC-type methods, and implicit Q-learning, implicit diffusion Q-learning [3] for Q-learning-type methods. Comparing DT with CQL and BC is a good starting point, but it could be more interesting to see those more advanced algorithms in these comparisons.

Author response:
We would like to express our gratitude to the reviewer for providing us with thorough feedback.

2. We appreciate the reviewers' suggestions to consider additional methods like IQL and TD3BC etc. We do not have a consensus on which algorithms can be considered SOTA in offline RL literature due to many confounding factors (stochasticity, dense/sparse settings etc.). For the sake of keeping our work relevant and broadly applicable, it is reasonable to consider algorithms which have become mainstream and are widely used. Our choice of algorithms (CQL, DT, BC) was driven by their widespread acceptance, ease of interpretation and compute budget. The simplicity of these algorithms, combined with our extensive understanding of them as a community, allows us to clearly trace performance fluctuations, which is crucial for our study's comprehensive analysis. In this way, our work provides valuable insights that enhance understanding of offline RL agents. We would like to highlight the compute and time concerns from the Common Response section to emphasize why having these agents made the most sense.

Due to these concerns, it was reasonable to leave out the diffusion based approach (it is yet to be widely adopted).

Here are some additional reasons why we did not study other methods.
- Numerous papers that are pivotal and investigative in nature (Engstrom et al. 2019, Andrychowicz et al. 2021, Ilyas et al. 2019) have showcased results on a small set of algorithms and environments:
Brandfonbrener et al. 2022 showed empirical results on BC, IQL and DT on 1 toy environment and 3 D4RL environments with standard configuration.
Kumar et al. 2022 showed results on only CQL and BC on 3 D4RL environments with standard configuration.
- GAIL (Ho et al. 2016) requires online reinforcement learning, diverging from our chosen pure offline RL setting where data remains static. TD3BC adds the BC objective into CQL. BC regularization is not as good as the one from CQL. When we are dealing with low-return data, we know it leans more on the BC objective. It is less interpretable.
- IQL could potentially bolster the perceived strength of our results. However, there's no consensus on its superiority over CQL. In many scenarios, CQL outperforms IQL (Kostrikov et al. 2021).
- Trajectory Transformer (Janner et al. 2021) employs planning in the learned model and is model based, unlike the model free methods which are the focus of our study. Additionally GATO  (Reed et al. 2022) is not open sourced, making DT the most appropriate choice in the paradigm.
- We know that %BC only considers a small subsample of the existing dataset. It throws out potentially useful data. For making a fairer comparison, it made sense for each of the agents to see the same amount of data.

Reviewer comment (Experiments):
It is known that BC does not perform well when the dataset contains not only high-return samples, but the performance of percentage BC in such settings other than Sec. 4.2 is not shown. It would be important to assess whether it is a robust and reliable method in the offline setting.

Author response:
We would like to express our gratitude to the reviewer for providing us with thorough feedback.

2. We appreciate the reviewers' suggestions to consider additional methods like IQL and TD3BC etc. We do not have a consensus on which algorithms can be considered SOTA in offline RL literature due to many confounding factors (stochasticity, dense/sparse settings etc.). For the sake of keeping our work relevant and broadly applicable, it is reasonable to consider algorithms which have become mainstream and are widely used. Our choice of algorithms (CQL, DT, BC) was driven by their widespread acceptance, ease of interpretation and compute budget. The simplicity of these algorithms, combined with our extensive understanding of them as a community, allows us to clearly trace performance fluctuations, which is crucial for our study's comprehensive analysis. In this way, our work provides valuable insights that enhance understanding of offline RL agents. We would like to highlight the compute and time concerns from the Common Response section to emphasize why having these agents made the most sense.

Due to these concerns, it was reasonable to leave out the diffusion based approach (it is yet to be widely adopted).

Here are some additional reasons why we did not study other methods.
- Numerous papers that are pivotal and investigative in nature (Engstrom et al. 2019, Andrychowicz et al. 2021, Ilyas et al. 2019) have showcased results on a small set of algorithms and environments:
Brandfonbrener et al. 2022 showed empirical results on BC, IQL and DT on 1 toy environment and 3 D4RL environments with standard configuration.
Kumar et al. 2022 showed results on only CQL and BC on 3 D4RL environments with standard configuration.
- GAIL (Ho et al. 2016) requires online reinforcement learning, diverging from our chosen pure offline RL setting where data remains static. TD3BC adds the BC objective into CQL. BC regularization is not as good as the one from CQL. When we are dealing with low-return data, we know it leans more on the BC objective. It is less interpretable.
- IQL could potentially bolster the perceived strength of our results. However, there's no consensus on its superiority over CQL. In many scenarios, CQL outperforms IQL (Kostrikov et al. 2021).
- Trajectory Transformer (Janner et al. 2021) employs planning in the learned model and is model based, unlike the model free methods which are the focus of our study. Additionally GATO  (Reed et al. 2022) is not open sourced, making DT the most appropriate choice in the paradigm.
- We know that %BC only considers a small subsample of the existing dataset. It throws out potentially useful data. For making a fairer comparison, it made sense for each of the agents to see the same amount of data.

Reviewer comment (Theory):
I am skeptical of the sparsified version of D4RL because it violates the Markov property of the reward function: the reward now depends on the whole trajectory rather than just the current state/action. And two trajectories may end at the same state but have significantly different total rewards, causing disagreement. I can see why this transformation would break CQL but not DT, and indeed CQL deterioriates much more on sparse D4RL than sparse Robomimic. So I think the results here do not convincingly show that CQL is worse than DT for sparse environments in general; on D4RL, this artificial transformation violates an assumption underlying the CQL algorithm, and on Robomimic, CQL was already worse than DT on Robomimic even when given dense rewards.

Author response:
1) **Sparse setting in a Markovian environment**. 
We agree that the existing D4RL sparse setting might not favor CQL due to non Markovian dynamics (similar to original DT study), we ran experimentation on Maze2D dense and split variants. Additionally we have experiments on Robomimic which also follows Markovian dynamics to back up our claims.

The following experiment was run on the sparse and dense variants of Maze as provided by the D4RL authors. It is worth noting that the Maze task explicitly requires trajectory stitching. As noted by Brandfonbrenfer et al. 2021, sequence modeling based approach would struggle in trajectory stitching. We observe this empirically here. Below we provide all the results sparse vs dense in addition to the experiments in paper to make comparison easier.

|                     | CQL           | DT (RTG=400)  | BC            |
|---------------------|---------------|---------------|---------------|
| Maze 2D Sparse      | $19.86(0.95)$ | $6.7(5.89)$   | $15.39(6.57)$ |
| Maze 2D Dense       | $49.62(7.79)$ | $27.9(6.85)$  | $38(5.23)$    |
| D4RL Sparse         | $83.64(0.51)$ | $103.15(0.77)$| $38.74(6.58)$ |
| D4RL Dense          | $40.78(7.88)$ | $76.6(1)$     | $38.74(6.58)$ |
| Robomimic Sparse    | $30(6.6)$     | $88.2(1.6)$   | $57.2(6)$     |
| Robomimic Dense     | $35.2(3.7)$   | $89.6(1.4)$   | $57.2(6)$     |
-----
We will adjust the claim to highlight that DT struggles on tasks requiring trajectory stitching and CQL should be a preferred choice. Due to shortage of time, we did not tune any parameters (incl. Experimenting with RTGs)

Reviewer comment (Experiments):
The experiment on stochasticity does not seem to perfectly answer the question “how good are these algorithms in stochastic environments?” because the data you are training on was collected with deterministic dynamics. So really what you are testing is “how well does a policy trained by these algorithms in a deterministic MDP transfer to a similar stochastic MDP?”. This may also be an interesting question to study, but I think that if you intend to answer the first question, the experiment is not entirely appropriate, and it would be best to repeat it with the training data also collected in a stochastic MDP.

Author response:
2) **Stochastic setting**. 
For comparison with existing stochastic experiments provided in the paper, we trained all the agents on deterministic data from D4RL, and introduced stochasticity during inference on stochastic (both params kept to 0.25).

|                   | CQL        | DT (RTG=400) | BC       |
|-------------------|------------|--------------|----------|
| Maze 2D Sparse    | 21.54(2.2) | 8.5(2.35)    | 4.62(3.91)|
| Maze 2D Dense     | 48.5(5.64) | 37.79(3.76)  | 33.72(5.4)|

Here also we see the trend where CQL emerges to be a preferred choice. The claims made in the paper remain intact.

One important concern which was raised is, we did not train agents in a stochastic setting. This is not a common setting in offline RL as the existing datasets are typically obtained in a deterministic environment. To address this concern, we trained a SAC agent to generate our own Maze 2D offline RL data in the same way as we generated humanoid with stochasticity parameters both set to 0.1 (prob=0.1, sigma=0.1) (trained for 2x longer to account for instability). 

The following results are obtained when all three agents were trained on stochastic data followed by inference in stochastic setting (both params set to 0.25)

|               | CQL         | DT (RTG=400) | BC        |
|---------------|-------------|--------------|-----------|
| Maze 2D Dense | 30.71(0.04) | 36.31(4.58)  | 32.4(6.88)|

We observed that DT outperforms CQL and BC in this scenario. This is an interesting finding showing that when all three agents are trained in a stochastic setting, CQL may or may not be the best choice when evaluated in a stochastic setting.

Reviewer comment (Novelty):
Some of the conclusions regarding BC are obvious (e.g. BC suffers more than offline RL when there is suboptimal/noisy data) or already widely known (e.g. BC works better than offline RL on human-generated demonstrations). It could provide more insight to include a hybrid method such as TD3-BC and show how the degree of regularization affects the behavior.

Author response:
3) While some of the conclusions regarding BC were known to the community (such as it typically thrives in expert data regime), our work provided multitudes of additional insights about BC which are novel (are not widely known or have not been shown empirically with this much thoroughness), we mention here a few points that can be considered novel about BC.
- The trend shown in Figure 1 and Figure 9 of %BC gives a deep insight about the behavior of DT as more high return and low return data is made more accessible, at a much more granular level not seen previously in other works.
- BC can be a preferred choice over CQL in sparse environments (especially in cases where rewards are not known)
- BC shows impressive performance on offline RL tasks, outperforming DT and CQL, when data is collected from human demonstrations. We have observed this extensively and have been observed in previous works.
- BC stays competitive when the task horizon is increased compared to other agents.

The main point we want to highlight is all of the claims are backed up by rigorous experimentation, providing researchers with reliable insights.

Reviewer comment (Writing):
What is a “high-quality continuous action space”? (near top of pg. 9) Does this just refer to the degree of determinism?

Author response:
4) By “high-quality continuous action space”, we are pointing out that if the offline RL data contains trajectories which are deemed high quality (medium-expert), then the drop in performance of DT in stochastic settings might be comparable to that of CQL as our findings show. We will rephrase this sentence to make it more clear.

Reviewer comment (Theory):
While the paper offers great empirical analysis, it would be still great to provide some technical/theoretical justifications regarding each empirical result shown in the paper, similar to [1], which will further strengthen the claim in the paper.

Author response:
We thank the reviewer for taking time to provide feedback. 

We addressed the theoretical justification concern in the common response section.

Reviewer comment (Experiments):
The empirical experiments are done in Robomimic and D4RL, but the D4RL tasks are only mujoco locomotion tasks. The authors should include experiments on more complex tasks such as Androit, Antmaze and vision-based tasks, e.g. vision-based drawer manipulation tasks in [1] and Atari games.

Author response:
We thank the reviewer for taking time to provide feedback. 

2) As requested by the reviewer, we are providing additional results on more complex tasks of D4RL such as Antmaze and Adroit. We have tons of results on Atari in Appendix (please see Section H) (due to lack of space in main text space). We were not able to add vision tasks due to reasons mentioned in the Common Response section.

|                   | CQL        | DT (RTG=400) | BC      |
|-------------------|------------|--------------|---------|
| Antmaze-umaze-v1  | 71.33(2.05)| 66.66(5.31)  | 62.6(6.8)|
| Pen-Human-v0      | 72.52(2.31)| 78.36(4.09)  | 85.68(6.77)|

- Antmaze also requires agents to explicitly stitch trajectories. Based on the empirical findings on this environment, we will highlight the trajectory stitching drawback with DT. 
- Results on Adroit corroborate our finding that BC is a superior choice when the data comes from human demonstrations followed by DT. A trend we have seen previously with Robomimic.

We hope this addresses all of the concerns of Reviewer pkaE.

Reviewer comment (Theory):
Please offer theoretical insights in each of the empirical findings where DT is more robust.

Author response:
We thank the reviewer for taking time to provide feedback. 

We addressed the theoretical justification concern in the common response section.

Reviewer comment (Experiments):
Please perform experiments in more complex tasks such as Androit, Antmaze and vision-based tasks, e.g. vision-based drawer manipulation tasks in [1] and Atari games.

Author response:
We thank the reviewer for taking time to provide feedback. 

2) As requested by the reviewer, we are providing additional results on more complex tasks of D4RL such as Antmaze and Adroit. We have tons of results on Atari in Appendix (please see Section H) (due to lack of space in main text space). We were not able to add vision tasks due to reasons mentioned in the Common Response section.

|                   | CQL        | DT (RTG=400) | BC      |
|-------------------|------------|--------------|---------|
| Antmaze-umaze-v1  | 71.33(2.05)| 66.66(5.31)  | 62.6(6.8)|
| Pen-Human-v0      | 72.52(2.31)| 78.36(4.09)  | 85.68(6.77)|

- Antmaze also requires agents to explicitly stitch trajectories. Based on the empirical findings on this environment, we will highlight the trajectory stitching drawback with DT. 
- Results on Adroit corroborate our finding that BC is a superior choice when the data comes from human demonstrations followed by DT. A trend we have seen previously with Robomimic.

We hope this addresses all of the concerns of Reviewer pkaE.