Paper: True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
As for now it is hard to consider LoRA to be part of the proposed method: hard to see the benefits of the LoRA contribution to sample efficiency and no catastrophic forgetting without an ablation on with and without LoRA. The contribution of the method seems to be the action normalization and some prompt design (qualitatively assessed by the authors in 4.4). Could you add experiments on these points?

Author response:
Q1: As for now it is hard to consider LoRA to be part of the proposed method: hard to see the benefits of the LoRA contribution to sample efficiency and no catastrophic forgetting without an ablation on with and without LoRA. The contribution of the method seems to be the action normalization and some prompt design (qualitatively assessed by the authors in 4.4). Could you add experiments on these points?

A1: Thanks for your valuable comments. **We want to point out that we use LoRA mainly for parameter efficiency instead of sample efficiency.** Even using LoRA, TWOSOME still has about 500M parameters to update. In contrast, the PPO network only has about 10K parameters to update, which is 50,000X less than TWOSOME. But TWOSOME still exhibits much better sample efficiency than PPO with the same amount of data, we attribute this to the prior knowledge of LLM, instead of LoRA. 

As mentioned in the first point of the above official comment, in the section of **Parameter-efficient Fine-tuning Architecture**, one of our main baselines, GLAM requires 8 NVIDIA A100 80GB GPUs to train their framework with a much smaller LLM, Flan-T5 780M without using LoRA. They report using 18880 GPU hours on Nvidia A100 80GB in total. Our model is 9X larger than theirs. **We estimate that at least 32 NVIDIA A100 80GB GPUs are required to conduct the experiments to train LLaMA-7B without using LoRA, otherwise, there is even no way to load the model and do a complete backpropagation.** Training LLaMA-7B without using LoRA is significantly far beyond most research labs’ ability, including us. It also shows the importance of our non-trivial parameter-efficient architecture, which enables the whole framework to run in a single A100 40G GPU. It greatly lowers the threshold for researchers to conduct the following research.

Reviewer comment (Novelty):
Paper mentions that (Carta et al, 2023) “focus on primitive actions in toy environments without rich semantics“. I am not sure actions are primitive in (Carta et al, 2023) but rather semantically described actions, e.g. “go forward”, “turn left”, “drop and toggle”. What makes them more “primitive” seems to be the environment they chose for experimenting (BabyAI vs Overcooked/VirtualHome).

Author response:
Q2: Paper mentions that (Carta et al, 2023) “focus on primitive actions in toy environments without rich semantics“. I am not sure actions are primitive in (Carta et al, 2023) but rather semantically described actions, e.g. “go forward”, “turn left”, “drop and toggle”. What makes them more “primitive” seems to be the environment they chose for experimenting (BabyAI vs Overcooked/VirtualHome).

A2: Yes, you are correct. That is one of the main reasons why we do not think BabyAI is a suitable environment to exhibit the performance of LLM-based agents. **We argue that LLM-based agents should be deployed in environments as close to daily life or their learned specific domains as possible, like SayCan with high-level daily skills (household robotics).**  Then they can make full use of the prior knowledge learned in the training process. If LLMs are not familiar with the observation and actions, not only will they fail to apply their learned prior knowledge but also dramatically slow down the training process for their tremendous parameters. It also emphasizes the importance of prompt design. 

As we mentioned in the above official comment, both the observation and action in BabyAI have relatively poor semantics. Here is an example prompt used by GLAM in BabyAI, “You carry a blue ball, You see a wall 4 steps forward, You see a wall 2 steps left, You see a grey key 1 step right and 1 step forward, You see a red box 2 steps forward”. It is rare for LLMs to see a corpus like this in the training process and we all know that even GPT-4 struggles with simple navigation tasks. It might partially explain why GLAM spent as much as 18880 GPU hours on Nvidia A100 80GB to get the results. 
LLM-based agents should not be applied to solve tasks in these environments under primitive settings. They are exactly the strong areas of RL agents. This is the reason why we insist on deploying our framework in daily environments with rich semantics for observation and actions.

Reviewer comment (Experiments):
Regarding the choice of environments. I am afraid there are too few fine-tuning tasks. I might be wrong, but the task generalization seems almost too-good from 4 tasks only. BabyAI was procedurally generated which would have enabled experimentation on more finetuning tasks.

Author response:
Q3: Regarding the choice of environments. I am afraid there are too few fine-tuning tasks. I might be wrong, but the task generalization seems almost too good from 4 tasks only. BabyAI was procedurally generated which would have enabled experimentation on more finetuning tasks.

A3: In the previous question Q2, we explained that we do not think BabyAI is a suitable environment to show the performance of our method. Here we provide another perspective.
Compared to our representative environments, BabyAI is a relatively easy environment, which can be totally solved by PPO as shown in [1]. According to Figure 4 and Table 1 shown in GLAM, even with a random policy, the agent can still have as high as 30% success rate.  In both of our two environments, the agent with random policy can never manage to finish the task with a success rate of 0. Even SayCan still has an extremely low success rate. We argue that BabyAI is not a suitable environment to show the performance of our methods. 

We believe our current representative environments and tasks are enough to show the robustness and generalization of our methods. Please refer to the second point of the above official comment, **About Evaluations on More Tasks and Environments**, for a clear explanation.  


[1] Chevalier-Boisvert M, Bahdanau D, Lahlou S, et al. Babyai: A platform to study the sample efficiency of grounded language learning[J]. arXiv preprint arXiv:1810.08272, 2018.

Reviewer comment (Novelty):
Can you elaborate on the difference between (Carta et al, 2019) and unnormalized action selection.

Author response:
Q4: Can you elaborate on the difference between (Carta et al, 2019) and unnormalized action selection.

A4: Sorry for the confusion. Due to the limitation of pages, we had to put the extended related work in Appendix A, where there are more discussions, especially the difference from previous works, like SayCan and GLAM. 

We assume the work that the reviewer refers to is GLAM (Carta et al, 2023). The main differences between TWOSOME without normalization and GLAM are the architecture and prompt design. 1) GLAM uses a much smaller encoder-decoder LLM, Flan-T5-780M, which requires 8 Nvidia A100 80GB GPUs to train. All TWOSOME agents use a 9X larger decoder-only LLM, LLaMA-7B, which only requires a single A100 40GB GPU to train, benefiting from our parameter-efficient framework. 2) GLAM focuses on simple primitive actions (turn left, turn right, go forward, etc.) evaluated in toy environments, BabyAI. These primitive actions have a similar number of tokens and less meaningful semantics, resulting in underutilizing the capabilities of LLMs, and failing to observe the impact of prompt design and address the unbalance over action space, resulting in additional instability and poor robustness. Though TWOSOME without normalization does not apply normalization, either, the design of observation and action prompts are more natural and robust.

Reviewer comment (Reproducibility):
Can you explain the PPO baseline: state/action space and architecture and initialization here (and in the main paper)? This would help in understanding the contribution of pretrained initialization.

Author response:
Q5: Can you explain the PPO baseline: state/action space and architecture and initialization here (and in the main paper)? This would help in understanding the contribution of pretrained initialization.

A5: Sorry for the confusion. Due to the limitation of pages, we had to put the description of state and action space in Appendix C and the architecture of PPO in Appendix D.1. We will move them to the main paper in the camera-ready version since there will be more space provided. PPO adopts symbolic observations, which consist of the positions and status of the agent and all the objects in the observations. Note that both two environments are partially observable. All the unseen objects will be masked. For action space, PPO adopts the same action space as TWOSOME. In the PPO method, both the actor and critic networks adopt a 3-layer MLP architecture with 64 neurons in each layer. PPO is initialized with orthogonal initialization.

Reviewer comment (Experiments):
Do you have insights on the use of embodied environments to **improve** LLMs, e.g. dealing with safety concerns? How to design such environments?

Author response:
Q6: Do you have insights on the use of embodied environments to improve LLMs, e.g. dealing with safety concerns? How to design such environments?

A6: Thanks for pointing this out. We have been very interested in deploying LLMs in various embodied environments to solve different complex tasks and improve LLMs themselves. Designing such environments from scratch might be costly and time-consuming. We believe that games are the perfect environments for LLMs to explore and self-improve. Games contain many rules and laws, which are carefully designed by the developers. These rules actually represent the values of humans, like what we should do, what we should not do, what is good, what is bad, where is safe and where is dangerous. 

There are many simulation games, which replicate certain situations in "real" life. **The Sims 4** might be one of the most famous games among them. Players can determine the appearances, personalities, and aspirations of the characters and need to help them pursue careers, develop skills, form relationships, and achieve life goals, like being a doctor, teacher and lawyer. This game provides tremendous alignment with reality. If we can make LLMs able to play the game well and manage to achieve life goals or good careers, then it means that LLMs have a good understanding of human society and that they can strictly follow the rules and laws to act as a good citizen according to the instructions of humans. We hope this may answer your questions on the safety concerns. 

There are already some works using LLMs to play Minecraft, like Voyager[1], GITM[2] and DEPS[3]. However, all of them heavily rely on GPT-4 to do the planning without updating and improving LLMs themselves. That is also the intuition of TWOSOME. Our method can online interact with the environments and keep improving LLMs with the experiences. We hope our method can be applied to more complex and realistic environments if more computational resources are provided.

[1] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).  
[2] Zhu, Xizhou, et al. "Ghost in the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory." arXiv preprint arXiv:2305.17144 (2023).  
[3] Wang, Zihao, et al. "Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents." arXiv preprint arXiv:2302.01560 (2023).

Reviewer comment (Experiments):
It would be nice to see how multi-modal text generation could be used to remove the assumption that the agent can extract a textual description of its state.

Author response:
Q7: It would be nice to see how multi-modal text generation could be used to remove the assumption that the agent can extract a textual description of its state.

A7: Yes, you are correct. A powerful VLM, like GPT-4V, can definitely greatly improve the performance and make our framework more complete. It will also facilitate the research in more complex environments as we mentioned in the last question, Q6. We are also thrilled to dig out GPT-4V’s ability, including image understanding and decision making, and imagine the infinite possibilities brought by GPT-4V.

Reviewer comment (Theory):
The overall approach of TWOSOME is to fine-tune a LLM agent (e.g., Llama-2-7B) with a RL algorithm (e.g., PPO). And, the unique contribution of TWOSOME can be seen as providing a normalization method for the probability distribution over text-based actions sampled from a LLM policy. This paper proposes two normalization methods: token normalization (Eq. 3) and word normalization (Eq. 4). Even though this paper proposes a promising direction towards fine-tuning a LLM agent with a RL algorithm, the main contribution seems rather marginal. The normalization method is about dividing the log probability of a varying length text-based action by the number of words (or the number of tokens). It seems to compensate a text action with larger number of words, but I am not sure that this kind of simple division is a proper way to normalize varying length text actions sampled from a LLM policy.

Author response:
Q2. Even though this paper proposes a promising direction towards fine-tuning a LLM agent with a RL algorithm, the main contribution seems rather marginal. It seems to compensate a text action with larger number of words, but I am not sure that this kind of simple division is a proper way to normalize varying length text actions sampled from a LLM policy.

A2: Thanks for your valuable comments.  We kindly remind the reviewer that the contribution of the whole framework may be underestimated. There are some previous similar works like SayCan and GLAM, but we managed to polish and optimize every module in the framework, resulting in high parameter-efficiency and better generalization and sample-efficiency. Please refer to the first point of the above official comment, **About Technical Contribution**, for a clear explanation.  

For the word normalization, we have introduced it in the previous question, Q1. We would not claim that our solution is the ultimate solution to the issue, but it turns out to be an easy-to-implement and feasible way to alleviate the issue, which is well acknowledged by the other three reviewers. Our method has shown significant improvement including both performance and stability according to the corresponding experiments and ablation studies. 

Not like the traditional natural language generation task where the output can be as long as several hundred tokens. The actions in traditional decision-making tasks are typically composed of a verb-object structure (e.g. pick up the apple), or Verb + Object + Preposition structure (e.g. put the apple on the table). They are usually within 20 tokens. Our experiments have covered the most common cases, which show that even in this range, the length of the prompts still has a significant influence on the joint probability of the actions and introduces much instability to the training process.

Reviewer comment (Experiments):
This paper shows that TWOSOME can learn the optimal policy in a simple environment like Overcooked. However, I am not sure that TWOSOME can properly learn optimal policy in a complex environment like VirtualHome. When showing the performance of TWOSOME in VirtualHome, the authors masks out unrelated actions to reduce the complexity of the action space.

Author response:
Q3. I am not sure that TWOSOME can properly learn optimal policy in a complex environment like VirtualHome. When showing the performance of TWOSOME in VirtualHome, the authors masks out unrelated actions to reduce the complexity of the action space.

A3: Sorry for the confusion. We intend to use action masking to prove that **compared to traditional RL methods, like PPO, it is more convenient for our method to mask actions and solve the tasks with a large action space**. When doing action masking, the network of PPO still needs to calculate the values of the masked actions, but we just do not use their values to form the policy in the final step. However, for our method, we just directly delete the masked actions from the action prompt list in the input, which also saves the computation for the values of these masked actions. Furthermore, it is also worth noting that our framework is also very flexible in extending the action space, by just directly add new actions to the action prompt list as the input, which is impossible for traditional RL methods. They need to train new agents from scratch with the new extended action space.

Moreover, we argue that struggling with large action space has been one of the main challenges for all decision-making methods, which is not a unique weakness of our method. Action masking is a very common trick to deal with large action space in decision-making tasks. The action space of all the environments is also a subset of the action space of reality, where these environments also mask out unrelated actions to reduce the complexity. It is fairly reasonable to mask invalid actions and unrelated items in the action space. 

Finally, we want to kindly remind the reviewer that even with the use of action masking, the difficulty of VirtualHome might be underestimated. Because we adopt a sparse reward setting, only when the agent successfully finishes the task will it receive +1 reward, as shown in Figure 5, even PPO with action masking, still fails to solve the task of Entertainment. In contrast, our method, TWOSOME with word normalization manages to learn the optimal policy, which already shows that our method has the ability in a complex environment.

Reviewer comment (Theory):
Regarding W1, how does the word normalization help LLM agents to learn better policy? What is the intuition of the word normalization?

Author response:
Q1. How does the word normalization help LLM agents to learn better policy? What is the intuition of the word normalization?

A1: For the intuition of the word normalization, we provided a detailed explanation in Section 4.1 in the main paper. Briefly speaking, **word normalization can provide a more accurate and reasonable initialized policy for the LLM agents to interact with environments**. So reasonable good actions, no matter whether they are long or short, always have higher probabilities to be sampled at the beginning of the training process, which would greatly accelerate the exploration process.

We observe that the probability of each token is always less than 1. Therefore, longer action prompts tend to have lower token-level probabilities, even though the longer action prompts may be more reasonable in the current situation. So we propose the token normalization to normalize the joint probability by the number of tokens (length). 

For example, if we have two actions, the first action has 20 tokens. To to simple, let the probability of each token in this action be as same as 0.8, which means this action is well-written and highly related to the task and the current observation. However, the joint probability of the action is 0.8^20 = 0.011. In contrast, the second action has only two tokens, with a probability of 0.2 for each token,  which means this action is much less relative to the task and the observation, and even the relation between these two tokens is quite weak. The joint probability of the second action is 0.2 * 0.2 = 0.04. Though each token in the first action is much higher than the second action, the first action finally has much fewer chances to be chosen, compared to the second action, which is not a desirable result. With the use of the token/word normalization by taking the lengths of the action into consideration, the joint probability of the action is still 0.8 and the joint probability of the second action is still 0.2 The distribution is much more reasonable. And the first action has a higher probability of being sampled.

However, we find the token normalization is slightly excessive. We observe that if a word is divided into several tokens, the first token usually has a relatively low probability, while the probabilities of the rest of the tokens tend to be remarkably high, which are often almost close to 100%. Therefore, it is more reasonable to regard the several tokens made up of one word as an integrated symbol. Then we propose our word normalization method to normalize the joint probabilities of actions by the number of words. Our experiments also demonstrate the same conclusion that TWOSOME without normalization is very unstable and fails to learn the optimal policy. Both TWOSOME with token normalization and word normalization have much better performance. And TWOSOME with word normalization converges much faster than token normalization since token normalization is slightly excessive.

In Appendix F, we also provide a detailed comparison and analysis of the probabilities generated by TWOSOME without normalization and with token/word normalization across all the tasks.

Reviewer comment (Experiments):
Regarding W2, without manual action masking in VirtualHome, how much scores can TWOSOME achieve?

Author response:
Dear reviewer fSyb:

We just added some new results in Appendix G.2 (on the last page of the paper), to show that **even without action masking, our method, TWOSOME with word normalization, can still manage to learn the optimal policy**, which shows the superior robustness and performance of our method and demonstrates that our method has the ability to deal with large action space.   

We hope these new results can solve the reviewer's doubts. As the deadline for the rebuttal period is approaching, is there any other clarification we can provide?

Reviewer comment (Experiments):
Recently, in RLHF, some enhanced RL algorithms such as Direct Policy Optimization (DPO) and Pair-wise Proximal Policy Optimization (P3O) have been proposed. Can these enhanced RL algorithms be applied to solving decision making tasks? And then, can we expect some performance improvements of TWOSOME?

Author response:
Q4. Recently, in RLHF, some enhanced RL algorithms such as Direct Policy Optimization (DPO) and Pair-wise Proximal Policy Optimizaition (P3O) have been proposed. Can these enhance RL algorithms be applied to solving decision making tasks? And then, can we expect some performance improvements of TWOSOME?

A4: To our understanding, compared to RLHF, both DPO and P3O attempt to directly use the preference data to train LLMs, saving the effort of training an accurate and explicit reward model or value function.  However, in the traditional decision-making setting, the reward function is already contained in the environment. We can directly use rewards to train our models instead of using preference pairs to infer the reward/value distribution. Therefore, DPO and P3O do not help in our standard RL decision-making settings.

Reviewer comment (Novelty):
The idea of querying joint probabilities of each valid action with LLM to form a policy is explored in previous works such as GLAM (Carta et al., 2023). The reviewer found, in the current version, the credit is not clearly given to authors of previous works.

Author response:
Thanks for pointing this out. **To our best knowledge, it is SayCan that first proposed to use joint probabilities of each valid action to form a policy**. And GLAM also acknowledges this point. Here is an excerpt from GLAM in Page 5, “Finally, Ahn et al. [2022] proposed to directly use the LLM to compute the (log) probability of each action by computing the conditional probability of each token in action given the prompt p”, where Ahn et al. [2022] refers to SayCan. We also mention this in our 4.1 Section that the way we used to form the policy is very similar to SayCan without the affordance function. We will add GLAM behind SayCan, since it also uses the same way to form the policy.

Reviewer comment (Experiments):
The reviewer has some concerns on the baselines used in the experiments. In Figure 5, aside from the ablations, the authors only consider classic PPO as a baseline. It seems unfair to directly compare an agent equipped with LLM with a classic trial-and-error PPO agent which doesn’t have access to the prior knowledge in LLM. There are many existing works that leverage LLM to form RL policies [1, 2, 3]. Particularly, Li [1] also uses virtualHome as test environments. Comparing approaches that also leverage knowledge in LLM could make the experimental section more convincing.

Author response:
Thanks for your pointing out. Please refer to the third point of the above official comment, **About Baselines Selection**, for a clear explanation. **Actually, we do compare our method with SayCan[2] and GLAM[3]**. Because we need to make some modifications and apply them to our tasks and domains, and we found that they also served as our ablation studies, for consistency, we call SayCan “TWOSOME without finetuning” and GLAM “TWOSOME without action prompt normalization” or directly “TWOSOME without normalization”. 

**As for [1], it is actually a purely supervised-learning-based method**. Instead of explicitly calculating the probabilities of each action to form standard RL policies, they directly generate actions using LLMs.  The loss they show in Equation 1, is exactly the same as the autoregressive loss. They just formulate the loss into a policy-like format. There is no difference between autoregressive supervised learning and imitation learning if actions are the only output of LLMs. And the authors also do not claim they are an RL-based method. In their paper, they also rely on pre-collected high-quality expert datasets. We do not think that they directly compete with our method. We can first apply their methods to do offline training if provided with a high-quality dataset, and then use our method to do online training by interacting with the environments.

Reviewer comment (Novelty):
The technical contribution of the paper is somewhat limited. Specifically, the idea of forming valid policy is proposed by previous works and LoRA for fine-tuning is a standard method. The review found the main new method is the proposed normalization approach. That to be said, the reviewer still considers the combination of existing methods might be valuable to our embodied AI community.

Author response:
We appreciate that the reviewer acknowledges our contribution to the embodied AI community. Please refer to the first point of the above official comment, **About Technical Contribution**, for a clear explanation.

Reviewer comment (Experiments):
As discussed in Weakness, comparing with methods that also leverage LLM for RL agent could make the experimental section more convincing.

Author response:
Thanks for your pointing out. Please refer to the third point of the above official comment, **About Baselines Selection**, for a clear explanation. **Actually, we do compare our method with SayCan[2] and GLAM[3]**. Because we need to make some modifications and apply them to our tasks and domains, and we found that they also served as our ablation studies, for consistency, we call SayCan “TWOSOME without finetuning” and GLAM “TWOSOME without action prompt normalization” or directly “TWOSOME without normalization”. 

**As for [1], it is actually a purely supervised-learning-based method**. Instead of explicitly calculating the probabilities of each action to form standard RL policies, they directly generate actions using LLMs.  The loss they show in Equation 1, is exactly the same as the autoregressive loss. They just formulate the loss into a policy-like format. There is no difference between autoregressive supervised learning and imitation learning if actions are the only output of LLMs. And the authors also do not claim they are an RL-based method. In their paper, they also rely on pre-collected high-quality expert datasets. We do not think that they directly compete with our method. We can first apply their methods to do offline training if provided with a high-quality dataset, and then use our method to do online training by interacting with the environments.

Reviewer comment (Writing):
In Eq (2), using $a_k$ in the summation is confusing.

Author response:
We greatly thank the reviewer for catching the typo. We have corrected it in the latest version.

Reviewer comment (Theory):
The paper does not provide a theoretical analysis of the proposed framework.

Author response:
Thanks for pointing it out. We indeed do not provide much theoretical analysis of the proposed framework because the whole framework is still basically under the traditional online actor-critic (PPO) framework, where the convergence of PPO should also apply. Though LLMs are usually transformer-based architecture, the last layer of LLMs is still an MLP layer to map the values generated by Transformer blocks to the probabilities of each token in the vocabulary.  We use the values to form the policies, which have no essential difference from the traditional RL agents, which usually have some CNN/MLP layers and use the output of the last MLP layer to form the policies. Finally, LoRA only updates part of the parameters, which may indeed prevent RL agents from learning the optimal policies. Empirically, LoRA shows superior performance not only in our work but also in tremendous CV/NLP works. We agree that it is non-trivial to provide a theoretical analysis of LoRA, especially in RL settings, but it is beyond the scope of this work.

Reviewer comment (Evaluation):
The paper does not discuss the potential limitations of the proposed framework.

Author response:
Thanks for pointing it out. Due to the limitation of pages, we do not provide a detailed analysis of the potential limitations. However, **we do mention our limitations in the final Discussion and Conclusion Section**, ‘However, our method suffers a major limitation that training a PPO agent from scratch is far faster and cheaper than finetuning an LLM. TWOSOME needs to feed all the valid actions to the LLMs for every action sampling, resulting in multiple times the amount of computation and a small batch size.’  Another limitation is that because we use an LLM as the backbone of our framework, we need to convert the observation into text and feed it into LLM, which inevitably causes information loss. The future work will be replacing the LLM with a VLM to directly use the image as input. We will put them in a separate section called Limitations and provide a more detailed discussion in the camera-ready version as there is more space provided.

Reviewer comment (Novelty):
The paper does not introduce anything novel, rather just combines existing components together to generalise to new tasks (apart from word normalisation which is fairly trivial).

Author response:
Thanks for your valuable comments. We kindly remind the reviewer that the contribution of the whole framework may be underestimated. There are some previous similar works like SayCan and GLAM, but we managed to polish and optimize every module in the framework, resulting in high parameter-efficiency and better generalization and sample-efficiency. Please refer to the first point of the above official comment, **About Technical Contribution**, for a clear explanation.

Reviewer comment (Experiments):
The paper does not compare to other LLM fine tuning baselines which generalise to new unseen tasks.

Author response:
Thanks for your valuable comments. To our best knowledge, only GLAM uses RL to train LLMs in embodied environments, which also serves as one of our main baselines. For the other supervised-learning-based methods, we do not think that they directly compete with our method. We can first apply their methods to do offline training if provided with a high-quality dataset, and then use our method to do online training by interacting with the environments. Please refer to the third point of the above official comment, **About Baselines Selection**, for a clear explanation.