Paper: Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICLR-2024)

Reviewer comment (Reproducibility):
The absence of shared code alongside the paper significantly hampers the research's transparency and the reproducibility of its findings. The explanation provided in Appendix A, particularly the vague descriptions in the concluding section, lacks the detailed guidance necessary for readers to independently replicate and verify the proposed method's effectiveness. Considering the empirical foundation upon which the method stands, it is imperative for the authors to release the corresponding code, ensuring that peers can thoroughly evaluate, validate, or even potentially improve upon the methodology.

Author response:
We promise that we will release our codes soon.

Reviewer comment (Experiments):
The comprehensive ablation studies provided are certainly insightful. Could the authors elaborate on their specific choice of using antmaze for these ablations (not MuJoCo), considering its characteristic sparse rewards and diverse objectives?

Author response:
One of the most important ADT's contribution is giving transformer-based models (DT) the stitching ability, referring to the capability to integrate suboptimal trajectories from the data. The Antmaze datasets consist of suboptimal trajectories, requiring the agent use portions of existing trajectories in order to solve a task, is widely considered to be the benchmark for evaluating the stitching ability [1]. Therefore, we choose to ablate the efficacy of each component of ADT in improving the stitching ability using Antmaze datasets.

### Reference

[1] Fu, Justin, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).

Reviewer comment (Novelty):
Integrating decision transformers into a hierarchical setting has already been studied in an earlier paper [*], especially corresponding to the goal-conditioned version. This paper wasn’t cited or discussed.

Author response:
Thanks for pointing this out. We have cited and discussed this paper and other DT Enhancements in Discussions section in our revised paper. ADT differs from HDT mainly from two aspects: (1) The high-level policy of ADT is more general that could output different kinds of prompts, including sub-goals, target values, latent skills and options. (2) For the low-level policy, we apply advantage weighted regression instead of conditioned prediction as in DT to enable stitching ability.

Reviewer comment (Evaluation):
The paper doesn’t dive into the analysis of the reason why the proposed approach outperforms or underperforms the baseline methods. For instance, when comparing with HRL baselines, it was simply mentioning that “Given that V-ADT and G-ADT is trained following the IQL and HIQL paradigm, respectively, the achieved performance nearing or inferior to that of IQL and HIQL is anticipated” - but why? Is it due to the generated subgoals or non-stationarity issues? The reasons behind the observations are more valuable to the community.

Author response:
Thanks for your suggestion! Our ablation studies are designed to explain the reasons behind the empirical results. But we totally agree that analysis of these results should be discussed more carefully in the paper.   

Our ablation studies reveal some interesting findings that might be able to explain why the overall performance of ADT falls short of IQL and HIQL. **Our main conjecture is that Transformer as a function approximator is hard to optimize for standard RL algorithms.** To verify this, in Section C in the appendix of the revised paper, we first implement an *oracle algorithm*, which distill the IQL policy using a transformer with supervised learning objective (V-ADT Oracle in Figure 6). The oracle algorithm matches the performance of IQL, suggesting that the transformer architecture is not the bottleneck. We then implement another baseline, named IQL-Trans , by replacing the MLP policy with a transformer policy for IQL. As shown in Figure 3, the performance of IQL-Trans (V-ADT w/o prompt in Figure 3) cannot match the original IQL, further supporting our conjecture. The advantage of ADT over IQL-Trans is mainly contributed to the joint optimization of hierarchical policies (the high-level policy optimizes the prompt and the the low-level policy is optimized based on the prompt), since this is the key difference between these two algorithms. To help readers clearly get this information, we provide the results of V-ADT, V-ADT Oracle, IQL-Trans and IQL in the table below:

|                           |     V-ADT      |   IQL-Trans    |  V-ADT Oracle  |       IQL       |
| :------------------------ | :------------: | :------------: | :------------: | :-------------: |
| antmaze-medium-play-v2    | 62.2 $\pm$ 2.5 | 50.6 $\pm$ 6.6 | 69.0 $\pm$ 1.8 | 71.2 $\pm$ 7.3  |
| antmaze-medium-diverse-v2 | 52.6 $\pm$ 1.4 | 38.6 $\pm$ 5.4 | 64.8 $\pm$ 6.5 | 70.0 $\pm$ 10.9 |
| antmaze-large-play-v2     | 16.6 $\pm$ 2.9 | 19.4 $\pm$ 3.6 | 50.0 $\pm$ 1.7 | 39.6 $\pm$ 5.8  |
| antmaze-large-diverse-v2  | 36.4 $\pm$ 3.6 | 5.0 $\pm$ 5.2  | 33.4 $\pm$ 5.3 | 47.5 $\pm$ 9.5  |

Finally, we note that there is still a performance gap between ADT and the oracle algorithm. This motivates the investigation of other techniques to improve  transformer-based decision models, which we leave as our future work.

Reviewer comment (Theory):
In V-ADT, does the high level provide prompt at every time step? If it does, it is arguably a hierarchical setting since there is no clear decomposition of a global goal or task. What is the reward function of the high level?

Author response:
Yes, we describe this in Figure 1. 



### Q: V-ADT is arguably a hierarchical setting since there is no clear decomposition of a global goal or task

A: The term "hierarchical" is mainly referred to the policy decomposition $\pi(a|s) = \int_{\rho\in \mathcal{P}} \pi(\rho | s) \pi(a|s,\rho) d \rho$  (Eq 3). The prompt is a general concept, includes but not limits to sub-goals used in HRL. Other choices includes returns-to-go (values), option, and skills. For example, return-conditioned prediction [1] can also be included as a special case of hierarchical policy framework. 

Although V-ADT does not decompose a global goal into explicit sub-goals, it operates hierarchically by having the high-level policy generate context-specific prompts (values in this case) which guide the decision-making of the low-level policy. This separation of prompt generation (high-level) and action determination (low-level) based on these prompts aligns with the fundamental concept of hierarchy in decision-making processes.



### Q: What is the reward function of the high level?

A: There is no reward function of the high-level in V-ADT. The low-level policy of V-ADT makes return-conditioned predictions as in [1]. The key innovation of V-ADT is the value prompt given by the high-level policy is learned from data.

Reviewer comment (Theory):
It’s not clear to me how the high level generates returns which are not covered in the offline dataset either. i.e., how could you avoid the issues you stated in Sec. 2

Reference: [*] Correia, A. and Alexandre, L.A., 2022. Hierarchical decision transformer. arXiv preprint arXiv:2209.10447.

Author response:
V-ADT can solve the motivating example. The value prompt used by V-ADT represents the maximum achievable value given the offline data. This value prompt then informs the V-ADT low-level policy to produce an action towards achieving the value. 

Given this in mind, let's consider the motivating example.  The dataset \( D \) contains three trajectories $a → b → c$ leading to a return of 0, $d → b → e$ leading to a return of 10, and another trajectory starting from *$a$* and leading to a return of 1. As for this particular example, the value prompt for $a$ is 10. To achieve this value, at state $a$, V-ADT takes a step towards $b$, as the other path would lead a worse trajectory with return 1.

Reviewer comment (Evaluation):
The experimental results presented in the paper appear to lack significance. For instance, in Table 2, the performance of G-ADT is nearly indistinguishable from that of the waypoint transformer. In Table 1, V-ADT demonstrates significant improvements over DT only in the antmaze environments, with results still falling short of IQL.

Author response:
### Q: The experimental results presented in the paper appear to lack significance

A: We respectfully disagree with this. V-ADT outperforms DT by **20.5%** on MuJoCo datasets, **227.5%** on Antmaze datasets, and **42.5%** on Kitchen datasets.  G-ADT  outperforms WT by **14.2%** on Antmaze datasets.   

The overall performance of ADT falls short of IQL could be attributed to the transformer architecture of the policy. In fact, it is still an open problem if Transformer is more efficient than standard architectures in single task RL. Previous studies show that MLP is competitive with and sometimes more effective than Transformer in offline RL benchmarks (see Table 1 of RvS[1] for example, where RvS-R outperforms DT on Antmaze datasets in the reward-conditioned behavior cloning setting). One conjecture is that high-capacity modles only make sense when we train on large and diverse datasets [2]. We would like to explore this conjecture in future works by applying ADT for multi-modal and multi-task domains.

Reviewer comment (Evaluation):
While the application of various RL losses to DT within a hierarchical framework has elements of novelty, the significance of these contributions may be undermined by the underwhelming experimental outcomes.

Author response:
### Q: The experimental results presented in the paper appear to lack significance

A: We respectfully disagree with this. V-ADT outperforms DT by **20.5%** on MuJoCo datasets, **227.5%** on Antmaze datasets, and **42.5%** on Kitchen datasets.  G-ADT  outperforms WT by **14.2%** on Antmaze datasets.   

The overall performance of ADT falls short of IQL could be attributed to the transformer architecture of the policy. In fact, it is still an open problem if Transformer is more efficient than standard architectures in single task RL. Previous studies show that MLP is competitive with and sometimes more effective than Transformer in offline RL benchmarks (see Table 1 of RvS[1] for example, where RvS-R outperforms DT on Antmaze datasets in the reward-conditioned behavior cloning setting). One conjecture is that high-capacity modles only make sense when we train on large and diverse datasets [2]. We would like to explore this conjecture in future works by applying ADT for multi-modal and multi-task domains.

Reviewer comment (Experiments):
Can more HRL environment experiments be shown?

Author response:
### Q: Can more HRL environment experiments be shown?

A: Our method is not designed specifically for HRL. Instead, we use hierachical framework to rethink transformer-based decision algorithms, and derive joint policy optimization algorithms from it. Our method is essentially an offline RL method and we follow standard benchmarks to prove the effectiveness of ADT.

Reviewer comment (Novelty):
The paper is missing key related works, such as prompt decision transformer studying the prompting mechanism for continuous control tasks in multi-task setting and hierarchical decision transformer, which has a similar motivation of using a hierarchical architecture. Prompting decision transformer for few-shot generalization: https://arxiv.org/pdf/2206.13499.pdf; Hierarchical decision transformer: https://arxiv.org/pdf/2209.10447.pdf.

Author response:
Thanks for pointing out this. We have cited and discussed these two papers and other DT Enhancements in Discussions section in our revised paper

Reviewer comment (Novelty):
Although the problem is quite solid, and the paper is trying to propose a general hierarchical framework to tackle the problem, the paper is still focusing on single-task learning, which is a relatively narrow scope.

Author response:
Actually, some of  the tasks in our experiments include several sub-tasks. For example, the kitchen datasets are collected based on the 9 degrees of freedom [Franka robot](https://www.franka.de/). The Franka robot is placed in a kitchen environment containing several common household items: a microwave, a kettle, an overhead light, cabinets, and an oven. The environment is a `multi-goal reaching` problem in which the robot has to interact with the previously mentioned items in order to reach a desired goal configuration. Detailed descriptions can be found at https://robotics.farama.org/envs/franka_kitchen/franka_kitchen/ and https://github.com/Farama-Foundation/d4rl/wiki/Tasks#frankakitchen. We leave multi-task learning on more benchmarks as our future work.

Reviewer comment (Writing):
The methodology is hard to follow, and the design choice is not fully discussed.

Author response:
To help understand our methodology better, we provide more detailed descriptions in Section 3 of the revision. As for the design choices, we ablate the effectiveness of the most important components of ADT including the efficacy of the prompt, the efficacy of the RL loss and the efficacy of tokenization in Section 5.4. If you could please point out the specific design choices that need to be discussed, we would happy to respond to them.

Reviewer comment (Writing):
The motivating example in Section 2.2.1 does not quite connect to the proposed methodology and leads to confusion. The example provided in section 2.2.1 is mainly about the data coverage issue. The proposed method seems to still learn Q, V, or goal from the offline dataset. How the proposed method can help solve the problem is still unclear.

Author response:
V-ADT can solve the motivating example. The value prompt used by V-ADT represents the maximum achievable value given the offline data. This value prompt then informs the V-ADT low-level policy to produce an action towards achieving the value. 

Given this in mind, let's consider the motivating example.  The dataset \( D \) contains three trajectories $a → b → c$ leading to a return of 0, $d → b → e$ leading to a return of 10, and another trajectory starting from *$a$* and leading to a return of 1. As for this particular example, the value prompt for $a$ is 10. To achieve this value, at state $a$, V-ADT takes a step towards $b$, as the other path would lead a worse trajectory with return 1.

Reviewer comment (Theory):
For the hierarchical policy, why condition on the current state is sufficient?

Author response:
We think a memoryless policy is sufficient for the Markov environment. However, we certainly agree that applying an auto-regressive model for the high-level policy as did in HDT is also possible. We would like to explore this design in our future work. Thanks for your suggestions.

Reviewer comment (Experiments):
Does the method’s performance heavily depend on the learned Q and V function from IQL?

Author response:
Yes, V-ADT requires joint optimization of high and low level policies. The output of the high-level policy, which is produced by IQL, significantly affects the performance of V-ADT. 

We want to emphasize that this does not mean that V-ADT is  simply reinventing IQL by replacing the policy with a transformer. Our ablation shows that leveraging the prompts from a high-level policy significantly boost the performance of V-ADT on some datasets.

Reviewer comment (Writing):
How the proposed method can help solve the example problem in section 2.2.1 (no trajectory a->b->c)?

Author response:
V-ADT can solve the motivating example. The value prompt used by V-ADT represents the maximum achievable value given the offline data. This value prompt then informs the V-ADT low-level policy to produce an action towards achieving the value. 

Given this in mind, let's consider the motivating example.  The dataset \( D \) contains three trajectories $a → b → c$ leading to a return of 0, $d → b → e$ leading to a return of 10, and another trajectory starting from *$a$* and leading to a return of 1. As for this particular example, the value prompt for $a$ is 10. To achieve this value, at state $a$, V-ADT takes a step towards $b$, as the other path would lead a worse trajectory with return 1.

Reviewer comment (Novelty):
How is the proposed method different from the hierarchical decision transformer?

Author response:
ADT differs from HDT mainly from two aspects: (1) The high-level policy of ADT is more general that could output different kinds of prompts, including sub-goals, target values, latent skills and options. (2) For the low-level policy, we apply advantage weighted regression instead of conditioned prediction as in DT to enable stitching ability.