Paper: Adapting LLM Agents Through Communication (ICLR-2024)

Reviewer comment (Experiments):
The authors proposes three different communication patterns which is nice, however, the experiments only study one pattern for each task. This makes it unclear on what are the pros and cons comparing these three communication patterns, whether each of them could generalize across different tasks and how to choose the right communication pattern. I feel it would be great to have more thorough comparison among the three patterns, and just with the baseline instruction tuning and prompting.

Author response:
Thank you very much for the constructive comments. We just open-sourced our code in an anonymous Github repo (https://github.com/AIAnonymous/LTC) to help the research community further explore this interesting topic. We'll de-anonymize the repo after the review process.
1. Communication patterns across Tasks

    To make the differences among patterns more clear, we reformulate the communication patterns as follows:
    - (1) **Single-agent Monologue** allows a single agent to generate trajectories in a self-talk style and get interactive feedback from environments; 
    - (2) **Multi-agent Dialogue** enables the multiple agents to interact with each other and external tools, and utilizes the reward signals provided by the environments; 
    - (3) **Teacher-student Dialogue** is a variant of multi-agent dialog in which the teacher agent can provide not only linguistic feedback but also non-linguistic reward signals.

    Different patterns are applied to tasks based on their difficulty and characteristics. In simulated Alfworld environments, the tasks are relatively easy. LLM agents can explore these tasks effectively. For instance, an agent trained with our single-agent Monologue communication achieves a 91% overall success rate. This rate even surpasses GPT-4's success rate of 89.6%. Therefore, there's no need to use multi-agent patterns with GPT-4 as a teacher in this scenario.

    However, as shown in the following table, the situation is different for other tasks. Using only the single-agent Monologue patterns to train the LTC agent shows limited improvements. In these cases, the GPT-4 agent performs much better than our Llama-7B agent. To address this, we designed multi-agent Dialogue and teacher-student Dialogue patterns. These are specifically for HotpotQA and GSM8k tasks. By leveraging GPT-4 agents in the loop, we aim to further improve our agents' training.

    | Method \ Task      |  GSM8K   |  HotpotQA  |  Alfworld |
    |--------------|--------|------------|-----|
    | GPT-4 prompting | 87.1   | 55.4  | 89.6 |
    | Llama-7B-Tuning | 37.7   |  28.2  | 78.4 |
    | Llama-7B-LTC (Monologue) | 39.6   | 31.0   | 91.0 |
    | Llama-7B-LTC (Dialogue)    | 41.3 |  33.2  | - |

Reviewer comment (Experiments):
The experiment is only done with the 7B model. It is not clear whether the method could apply to smaller models, and more importantly whether it could scale up and how much improvement it could bring to models of larger size.

Author response:
2. **Scaling up for smaller and larger LLMs**

    To further investigate the scaling-up effectiveness of our method, we conduct extra experiments based on smaller (1B) and larger (13B) models on HotpotQA, the results are shown below: 

    | Model           | Method                                             | EM score |
    |----------------------|----------------------------------------------------|----------|
    | PaLM-540B       | CoT                                            | 29.4     |
    |                          | CoT-SC                                         | 33.4     |
    |                          | ReAct                                          | 27.4     |
    |                          | ReAct → CoT-SC                                     | 35.1     |
    | GPT3-175B      | ReAct                                              | 30.8     |
    | PaLM-62B        | ReAct-Tuning                                       | 32.6     |
    |                          | CoT-Tuning                                         | 25.2     |
    | PaLM-8B          | ReAct-Tuning                                       | 25.0     |
    |                          | CoT-Tuning                                         | 14.0     |
    | Llama-1B[*](https://github.com/jzhang38/TinyLlama)     | ReAct-Tuning                                       | 12.0     |
    |                          | LTC                                                | 15.4     |
    | Llama-7B        | ReAct-Tuning                                       | 28.2     |
    |                          | LTC                                                | 33.2     |
    | Llama2-13B   | ReAct-Tuning                                       | 33.8     |
    |                          | LTC                                                | **35.8** |


    It shows that our method with a Llama-2-13B can even surpass the huge PaLM-540B by 0.7%, which is 41x larger. The tiny Llama model with 1B parameters can also benefit from our method, outperforming its tuning baseline. The improvement of our method is consistent when scaling up.

Reviewer comment (Experiments):
Have you tried the three patterns across tasks, or they are only tested on the specific task?

Author response:
Thank you very much for the constructive comments. We just open-sourced our code in an anonymous Github repo (https://github.com/AIAnonymous/LTC) to help the research community further explore this interesting topic. We'll de-anonymize the repo after the review process.
1. Communication patterns across Tasks

    To make the differences among patterns more clear, we reformulate the communication patterns as follows:
    - (1) **Single-agent Monologue** allows a single agent to generate trajectories in a self-talk style and get interactive feedback from environments; 
    - (2) **Multi-agent Dialogue** enables the multiple agents to interact with each other and external tools, and utilizes the reward signals provided by the environments; 
    - (3) **Teacher-student Dialogue** is a variant of multi-agent dialog in which the teacher agent can provide not only linguistic feedback but also non-linguistic reward signals.

    Different patterns are applied to tasks based on their difficulty and characteristics. In simulated Alfworld environments, the tasks are relatively easy. LLM agents can explore these tasks effectively. For instance, an agent trained with our single-agent Monologue communication achieves a 91% overall success rate. This rate even surpasses GPT-4's success rate of 89.6%. Therefore, there's no need to use multi-agent patterns with GPT-4 as a teacher in this scenario.

    However, as shown in the following table, the situation is different for other tasks. Using only the single-agent Monologue patterns to train the LTC agent shows limited improvements. In these cases, the GPT-4 agent performs much better than our Llama-7B agent. To address this, we designed multi-agent Dialogue and teacher-student Dialogue patterns. These are specifically for HotpotQA and GSM8k tasks. By leveraging GPT-4 agents in the loop, we aim to further improve our agents' training.

    | Method \ Task      |  GSM8K   |  HotpotQA  |  Alfworld |
    |--------------|--------|------------|-----|
    | GPT-4 prompting | 87.1   | 55.4  | 89.6 |
    | Llama-7B-Tuning | 37.7   |  28.2  | 78.4 |
    | Llama-7B-LTC (Monologue) | 39.6   | 31.0   | 91.0 |
    | Llama-7B-LTC (Dialogue)    | 41.3 |  33.2  | - |

Reviewer comment (Experiments):
How many instruction data by GPT 3/4 (as mentioned in section 4.2) are used?

Author response:
3. **Instruction data**

    For Alfworld datasets, we collect 3.5k ReAct trails on its training set and only use the 1.9k successful trails as the instruction data to fine-tune the agents. For HotpotQA, we use 3.5k successful ReAct trail data from the training set. For GSM8k, we use 7.5k examples with CoT as the instruction fine-tuning data, the same scale as its own training data. We just open-sourced our code in an anonymous Github repo (https://github.com/AIAnonymous/LTC) to help the research community further explore this interesting topic. The instruction finetuning data are also included in this repo for reproducibility.

Reviewer comment (Experiments):
The ablation in Fig 5 doesn't seem to disentangle the contributions of exploration and "communication". In particular, performing exploration and receiving non-linguistic feedback (e.g., in the environment) doesn't seem to be evaluated. And on the other hand, training with static instruction-tuning data augmented with "communication" traces sampled using the same method as LTC (without sampling agent actions, but using the static demonstrations instead) is not evaluated.

Author response:
Experimental setup: performing exploration and receiving non-linguistic feedback
With our new formulation of the communication patterns, our work mainly targets on studying agent adaptation algorithms in various environments (i.e., single/multi-agent setup). We think the reviewer wants to ask 1. the comparison of “single-agent” v.s. “multi-agent” setup in one particular task. 2. reinforce the learning algorithm with environment reward v.s. (environment reward + feedback signals from other agents).

For “single-agent” v.s. “multi-agent”, we add the experiments on HotpotQA (in Table 2) to show the the comparison between single-agent monologue and multi-agent dialogue, which shows that using Multi-agent Dialogue can further improve the performance: 
| Method      |  EM score  |
|--------------|--------|
| Llama-7B-Tuning  |  28.2  | 
| Llama-7B-LTC (Single-agent Monologue) | 31.0   | 
| Llama-7B-LTC (Multi-agent Dialogue)   |  **33.2**  | 

For environment reward v.s. (environment reward + feedback signals from other agents), we add the experiments on GSM8k (in Table 3) to show the comparison between single-agent monologue and teacher-student dialogue, which shows that using teacher-student Dialogue can further improve the performance by providing the direct reward signals from the GPT-4 agent: 
| Method      |  Acc  |
|--------------|--------|
| Llama-7B-Tuning  |  37.7  | 
| Llama-7B-LTC (Single-agent Monologue) | 39.6   | 
| Llama-7B-LTC (Teacher-student Dialogue)   |  **41.3**  | 


**Experimental setup: training with static instruction-tuning data**

We want to confirm that you are suggesting two more baselines: 1. SFT on (Instruction-tuning dataset + static few-shot dataset); 2. SFT on (instruction-tuning dataset + communication traces). 
For the 1st baseline, we don’t see the motivation of adding “static demonstration” in the prompt of the training set, since our method does not require few-shot examples as prompt.
For the 2nd baseline, we present the comparison results in Figure 5, the “w/o PPO” curve can be an example of SFT on communication traces since it uses the exploration traces as the training data and only does SFT on the text part of the data. Our takeaway from the Fig 5 is that what really matters is the communication traces generated by the agent itself, which is better than the SFT data generated by other powerful LLMs.

Reviewer comment (Novelty):
I would suggest renaming the approach away from "through communication". This phrase is very vague, and there really isn't any true communication happening here at all: the learning setting is to just prompt the model(s) to replicate what a dialogue might look like in the domain. It is much more reminiscent of work on distillation and learning from oracles, like DAgger (as briefly mentioned in a footnote on page 7).

Author response:
**Rename the approach**

We appreciate the reviewer's suggestion. In our multi-agent framework, we attribute performance improvement primarily to effective communication, where multiple agents exchange information throughout the process without manual intervention. This is evident in our study, where the student agent, by communicating with teacher agents or the environment, learns more effectively than when limited to static training data.

To further illustrate the role of communication, we introduced the Chameleon game environment in our experiments. Detailed in section 4.2 of our updated paper, the Chameleon game involves two roles: chameleon and non-chameleon. In our setup, all players are played by the Llama2-7B model during training, but in testing, one player is randomly selected to be played by our trained agent, with the others played by GPT-4. The table below showcases the winning rates of our trained agent against GPT-4 players:

| Method \ #players       | n=3    | n=4    | n=5    |  overall |
|--------------|--------|--------|--------|---------|
   | Tuning | 20.8   | 20.3   | 23.8   | 21.9    |
| LTC (multi-agent dialogue)   | **22.9** | **23.4** | **27.5** | **25.0**  |

The results show that our LTC method surpasses instruction tuning baselines by 3.1% in overall winning rates. This underscores the importance of communication in the Chameleon game, which demands bluffing, deduction, and strategic interaction between agents and their environment for success.

Reviewer comment (Experiments):
Framing this as continual learning also seems wrong. Fine-tuning with LTC is performed on some held-out set of examples from these datasets' training sets, right? So how is this continual?

Author response:
**Distinguishing from continual learning**

Yes, we should not frame our methods as continual learning, so we rewrite all parts in the paper that mention continual learning to avoid misleading. With our new formulation of the communication patterns, our current paper focuses on training the LLM agents for single-agent and multi-agent environments.

Reviewer comment (Writing):
Minor nitpick, but the whole learning setup is RL, not just the training phase, as the exploration part of the proposed approach is certainly part of a general RL framework.

Author response:
**The training phase**

Sure, we’ve renamed the Training Phase as the Updating Phase in the text and figures. Now the iterative training is a standard reinforcement learning pipeline, which has the exploration phase and the updating phase.

Reviewer comment (Presentation):
There seems to be a bug in Fig 7 with the "question".

Author response:
**The bug in Figure 7**

Thanks for reminding us, the question text is forgotten to replace when using the template of Fig3, we’ve updated it with the correct question text.

Reviewer comment (Writing):
The terminology "generate trajectories in a self-talk style" is unclear.

Author response:
**Ambiguous terminology**

These comments are very helpful in our revision, now we have updated our draft based on your suggestions. The core change is the definitions of the communication patterns. To make it more clear,  we reformulate the communication patterns as follows:
- (1) **Single-agent Monologue** allows a single agent to generate trajectories in a self-talk style and get interactive feedback from environments; 
- (2) **Multi-agent Dialogue** enables the multiple agents to interact with each other and external tools, and utilizes the reward signals provided by the environments; 
- (3) **Teacher-student Dialogue** is a variant of multi-agent dialog in which the teacher agent can provide not only linguistic feedback but also non-linguistic reward signals.

Reviewer comment (Experiments):
In the experimental section, some existing methods are introduced and compared against without providing intuition on why the experiment is performed (unless the reader is very familiar with the existing approaches). For example, what data are ReAct-Tuning and CoT-Tuning trained on? There is a self-reference to Section 4.2.

Author response:
**The actual steps for training**

There are only two steps starting from the pre-trained LLMs(I.e, Llama): 
- 1. We use the collected task-specific instruction fine-tuning data to fine-tune the Llama to help it learn the instruction-following ability (like ReAct/CoT). And the output models are our Llama-Tuning baseline in our experiments.
- 2. We put the fine-tuned agents in our communication patterns to explore the environments and collected the traces for training iteratively.

Reviewer comment (Experiments):
Model Comparisons: The LTC method's performance is compared to that of models with and without tuning. However, the paper states that the combined method of ReAct and CoT-SC surpasses LTC by 1.9%. This suggests that while LTC has strengths, there may be specific configurations of existing methods that outperform it, which could be a point of concern regarding the robustness and superiority of LTC​1​.

Author response:
We thank the reviewer for noting the comparison between LTC and other methods such as ReAct and CoT-SC in Table 2. We would like to point out that results on ReAct and CoT-SC are reported based on the PaLM (540B) model, while LTC is on Llama (7B). To further investigate the scaling-up effectiveness of our method, we conduct extra experiments based on smaller (1B) and larger (13B) models on HotpotQA, the results are shown below: 

| Model           | Method                                             | EM score |
|----------------------|----------------------------------------------------|----------|
| PaLM-540B       | CoT                                            | 29.4     |
|                          | CoT-SC                                         | 33.4     |
|                          | ReAct                                          | 27.4     |
|                          | ReAct → CoT-SC                                     | 35.1     |
| GPT3-175B      | ReAct                                              | 30.8     |
| PaLM-62B        | ReAct-Tuning                                       | 32.6     |
|                          | CoT-Tuning                                         | 25.2     |
| PaLM-8B          | ReAct-Tuning                                       | 25.0     |
|                          | CoT-Tuning                                         | 14.0     |
| Llama-1B[*](https://github.com/jzhang38/TinyLlama)     | ReAct-Tuning                                       | 12.0     |
|                          | LTC                                                | 15.4     |
| Llama-7B        | ReAct-Tuning                                       | 28.2     |
|                          | LTC                                                | 33.2     |
| Llama2-13B   | ReAct-Tuning                                       | 33.8     |
|                          | LTC                                                | **35.8** |


It shows that our method with a Llama-2-13B can even surpass the huge PaLM-540B by 0.7%, which is 41x larger. The tiny Llama model with 1B parameters can also benefit from our method, outperforming its tuning baseline. The improvement of our method is consistent when scaling up.

Reviewer comment (Experiments):
Scope of Evaluation: While LTC is shown to perform well across three tasks, the evaluation might still be limited in scope. The paper hints at future work to explore more diverse communication patterns and involve communication with humans. This suggests that the current evaluation may not fully capture the LTC's performance in more varied or complex interactive settings​ such as ScienceWorld, and Mind2Web.

Author response:
We conducted some extra experiments on a more challenging multi-agent environment setup (i.e., Chameleon Game). Preliminary results are updated in the paper ( pls see section 4.2 for details).
There are two roles in the [Chameleon](https://github.com/Farama-Foundation/chatarena) game: chameleon and non-chameleon. The topic of the secret word will be first revealed to all the players. Then the secret word will be revealed to non-chameleons. Non-chameleons try to identify the chameleon without giving away the secret word, while the chameleon tries to blend in and guess the word. 
In our training, all the players are played by the same Llama2-7B model. While in the testing, to get the winning rate of our trained agent against GPT4, only 1 player is randomly picked to use our trained agent as the backend, and other players are played by GPT4. The following results are their winning rate against GPT-4 players:

| Method \ #players   | n=3    | n=4    | n=5    |  overall |
|-----------|--------|--------|--------|---------|
| Tuning | 20.8   | 20.3   | 23.8   | 21.9    |
| LTC    | **22.9** | **23.4** | **27.5** | **25.0**  |


As shown above, LTC outperforms the instruction tuning baselines by 3.1% overall winning rates. To further clarify, we aren’t trying to demonstrate that multi-agent setup is superior to single-agent setup in experiments, but we show that “learning through communication” allows agents to be online-trainable in various complex single/multiple agent environments through its conversation data and feedback signals.

Reviewer comment (Experiments):
Generalization to Human Interaction: The paper outlines future work to involve communication with humans during the learning process, which is not covered in the current evaluation. This omission indicates that the paper does not address the challenge of human-agent interaction, which is critical for practical applications of LLM agents.

Author response:
The integration of human-in-the-loop interactions with the LTC agent could significantly enhance the model's adaptability and learning process, particularly in terms of customization and personalization. By directly incorporating human feedback, LTC can tailor its responses more effectively to individual user preferences and requirements. This approach not only ensures that the technology adapts to the specific needs and contexts of different users but also improves user engagement and satisfaction. Such personalized interaction is crucial for the broad applicability of LTC in diverse practical scenarios, where unique user experiences and customized responses are highly valued.

Reviewer comment (Novelty):
Minor point: References that can be helpful for discussion:  
1) ScienceWorld: Is your Agent Smarter than a 5th Grader?  
2) Mind2Web: Towards a Generalist Agent for the Web  
3) AgentBench: Evaluating LLMs as Agents  
4) SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks  
5) ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models.

Author response:
Thanks for providing these helpful References, we’ve add them into our discussion of related works

Reviewer comment (Theory):
The LTC method relies heavily on predefined communication patterns. Have the authors considered how LTC might generalize to tasks that require more flexible or less structured forms of communication?

Author response:
Great question. We explored recent open source multi-agent libraries (i.e., AutoGen, ChatArena, Ai-Town etc), however most of them require users to predefine the communication patterns as well, while some of them define some randomness in the environment to allow the agent to communicate with another random agent (i.e., AI-town). We are actively exploring to build a large-scale multi-agents environment for simulating general collaboration and competition tasks between random-agents. We are hoping such a sandbox system could be a good testbed for comparing different LTC strategies.

Reviewer comment (Evaluation):
Given that certain combined methods like ReAct and CoT-SC have outperformed LTC, what are the authors' perspectives on the limitations of LTC in its current form? Are there specific enhancements they are considering to improve upon these existing methods?

Author response:
As shown above, we have demonstrated that by scaling up LTC to Llama2-13B, we can achieve superior results, surpassing even the larger PaLM-540B combined methods. This underscores the scalability and potential of LTC. Actually our LTC is orthogonal to these promising methods like ReAct and CoT, which focus on prompting and we are focusing on training. So the integration of our methods with these advanced prompting methods to improve the communication effectiveness is what we are investigating to further enhance our framework.
Of course there are also some limitations of LTC:
- Cost: computation resource requirement would be huge if we allow personal assistant agents to be trainable (alternative solution could be hosting LTC agents on personal edge device) 
- Stability: LTC adopts an RL approach for iterative learning. Thus it doesn’t guarantee to procedure a better-performance model after every epoch of training.

Reviewer comment (Theory):
Multi-agent interactions highlighted in the paper are not properly formulated and motivated.

Author response:
2. _____Multi-agent interactions are not properly formulated_____

    To further highlight the multi-agent interaction and make it more clear, we reformulate the communication patterns as follows:
    - (1) **Single-agent Monologue** allows a single agent to generate trajectories in a self-talk style and get interactive feedback from environments; 
    - (2) **Multi-agent Dialogue** enables the multiple agents to interact with each other and external tools, and utilizes the reward signals provided by the environments; 
    - (3) **Teacher-student Dialogue** is a variant of multi-agent dialog in which the teacher agent can provide not only linguistic feedback but also non-linguistic reward signals.

Reviewer comment (Experiments):
Authors are not considering any environments that require multi-agent interactions. This reviewer is not convinced that the paper adds value in terms of communication between multiple agents that would lead to better performance.

Author response:
1. ____The multi-agent environments____

    Thanks for your suggestion. We conducted some extra experiments on a multi-agent environment setup (i.e., Chameleon Game). Preliminary results are updated in the paper ( pls see section 4.2 for details). There are two roles in the [Chameleon](https://github.com/Farama-Foundation/chatarena) game: chameleon and non-chameleon. The topic of the secret word will be first revealed to all the players. Then the secret word will be revealed to non-chameleons. Non-chameleons try to identify the chameleon without giving away the secret word, while the chameleon tries to blend in and guess the word. 
    In our training, all the players are played by the same Llama2-7B model. While in the testing, to get the winning rate of our trained agent against GPT4, only 1 player is randomly picked to use our trained agent as the backend, and other players are played by GPT4. The following results are their winning rate against GPT-4 players:
    
    | Method \ #players       | n=3    | n=4    | n=5    |  overall |
    |--------------|--------|--------|--------|---------|
    | Tuning | 20.8   | 20.3   | 23.8   | 21.9    |
    | LTC    | **22.9** | **23.4** | **27.5** | **25.0**  |


    As shown above, LTC outperforms the instruction tuning baselines by 3.1% overall winning rates. To further clarify, we aren’t trying to demonstrate that multi-agent setup is superior to single-agent setup in experiments, but we show that “learning through communication” allows agents to be online-trainable in various complex single/multiple agent environments through its conversation data and feedback signals.

Reviewer comment (Reproducibility):
Authors have not provided the code and hence the results are not reproducible.

Author response:
Thank you very much for the constructive comments. We just open-sourced our code in an anonymous Github repo (https://github.com/AIAnonymous/LTC) to help the research community further explore this interesting topic. We'll de-anonymize the repo after the review process.

Reviewer comment (Theory):
According to the provided discussion the only difference between dialogue and analogue is in analogue, teacher roles can directly provide reward signals and new examples. What is the significance of this distinction?

Author response:
3. ____The difference between Dialogue and Analogue____ 
	
    As shown in our new multi-agent formulations, both the second and third patterns are designed for dialogue between multiple agents. The difference is that the agents in the second pattern can only receive the rewards provided by the environments, while the Teacher-student Dialogue patterns enable the teacher agent to directly give the reward to the student.

Reviewer comment (Theory):
How does this approach extend to multi-agent environments?

Author response:
1. ____The multi-agent environments____

    Thanks for your suggestion. We conducted some extra experiments on a multi-agent environment setup (i.e., Chameleon Game). Preliminary results are updated in the paper ( pls see section 4.2 for details). There are two roles in the [Chameleon](https://github.com/Farama-Foundation/chatarena) game: chameleon and non-chameleon. The topic of the secret word will be first revealed to all the players. Then the secret word will be revealed to non-chameleons. Non-chameleons try to identify the chameleon without giving away the secret word, while the chameleon tries to blend in and guess the word. 
    In our training, all the players are played by the same Llama2-7B model. While in the testing, to get the winning rate of our trained agent against GPT4, only 1 player is randomly picked to use our trained agent as the backend, and other players are played by GPT4. The following results are their winning rate against GPT-4 players:
    
    | Method \ #players       | n=3    | n=4    | n=5    |  overall |
    |--------------|--------|--------|--------|---------|
    | Tuning | 20.8   | 20.3   | 23.8   | 21.9    |
    | LTC    | **22.9** | **23.4** | **27.5** | **25.0**  |


    As shown above, LTC outperforms the instruction tuning baselines by 3.1% overall winning rates. To further clarify, we aren’t trying to demonstrate that multi-agent setup is superior to single-agent setup in experiments, but we show that “learning through communication” allows agents to be online-trainable in various complex single/multiple agent environments through its conversation data and feedback signals.