Paper: Lemur: Harmonizing Natural Language and Code for Language Agents (ICLR-2024)

Reviewer comment (Presentation):
As mentioned in Introduction, the paper has offered valuable insights on synergy, but it is unclear what the insights exactly are. I would suggest clearly presenting the insights instead of letting readers find where is the insights across the paper.

Author response:
We appreciate the detailed feedback you provided on our writing and clarity. Based on these suggestions, we have highlighted the most important insights about harmony ability with specific descriptions in the introduction. Additionally, we have added error analysis as well as additional findings and insights.

Reviewer comment (Writing):
Minor: In Figure 2, the capitalizations are not consistent. (Use->,  run->); Section 4.5: mapp -> map; Section 4.5: intermm.

Author response:
We have corrected the typos in our revised version. Thanks a lot for pointing them out!

Reviewer comment (Experiments):
Why is a large proportion of the pre-training data is in Python?

Author response:
We thank you for your valuable questions on our technical details!

- The data mixture strategy is a complex trade-off between the model size, the vanilla performance of the model, data quality, learning efficiency, catastrophic forgetting etc.
- To roughly estimate a good code-to-text ratio for composing the training corpora, we conducted a continue pre-training study with Llama on a set of different ratios and found that the ratio of 10:1 is an efficient ratio for the Llama model to transfer from text to text-code balance, which aligns with the findings of recent work like CodeLlama.
- Among all programming languages, we focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.) because agent models are often executed in interactive scenarios. Unlike compiled languages like C++, the interpreted nature of scripting languages allows immediate execution and easy modification, which is essential for dynamic interaction in language agent scenarios. 
- Due to computational limits, it is difficult for us to conduct comprehensive experiments and perform systematic studies on this scale. However, we hope that our open-sourced effective settings and training checkpoints can benefit the community for continual exploration.
- We believe that a method to predict the optimal continue-pre-training data mixture ratio for a pair of domains to maximize their performance would be very meaningful and interesting, and it is still an open research question for the current large language model.

We have added this explanation in the appendix of our revised version. Thanks for your valuable question!

Reviewer comment (Experiments):
Is the harmonization controlled by the text-to-code ratio? How did you come up with the idea of setting a ratio of 10:1?

Author response:
We thank you for your valuable questions on our technical details!

- The data mixture strategy is a complex trade-off between the model size, the vanilla performance of the model, data quality, learning efficiency, catastrophic forgetting etc.
- To roughly estimate a good code-to-text ratio for composing the training corpora, we conducted a continue pre-training study with Llama on a set of different ratios and found that the ratio of 10:1 is an efficient ratio for the Llama model to transfer from text to text-code balance, which aligns with the findings of recent work like CodeLlama.
- Among all programming languages, we focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.) because agent models are often executed in interactive scenarios. Unlike compiled languages like C++, the interpreted nature of scripting languages allows immediate execution and easy modification, which is essential for dynamic interaction in language agent scenarios. 
- Due to computational limits, it is difficult for us to conduct comprehensive experiments and perform systematic studies on this scale. However, we hope that our open-sourced effective settings and training checkpoints can benefit the community for continual exploration.
- We believe that a method to predict the optimal continue-pre-training data mixture ratio for a pair of domains to maximize their performance would be very meaningful and interesting, and it is still an open research question for the current large language model.

We have added this explanation in the appendix of our revised version. Thanks for your valuable question!

Reviewer comment (Theory):
It seems that pre-training takes the responsibility to gain the coding ability, and the supervised fine-tuning takes the responsibility to gain the natural language ability, while it is vague how the proposed model balance these two abilities.

Author response:
- The strategy we use to balance the two abilities involves two-stage training. First, we train a good text model LLaMA-2-70B on code-intensive tasks, which aims to enhance its coding abilities. This gives Lemur-70B. Second, we finetune Lemur-70B to improve its instruction-following capabilities on both text and code scenarios.
- From the results we present in Table 3 (which we also copy here for easier reference), it can be seen that Lemur-70B outperforms LLaMA-2-70B in code tasks, but maintains a good balance in text tasks. Additionally, thanks to its instruction-following capability, Lemur-70B-Chat achieves better performance in all scenarios.


| Model | Text |  |  | Code |  |  |  |  | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | QA | Reason | Math | Python |  | SQL | MCode | DS |  |
|  | MMLU | BBH | GSM8K | HumanEval | MBPP | Spider | MultiPL-E | DS-1000 |  |
| LLaMA-2-70B | 68.9 | 51.2 | 56.8 | 30.5 | 45.4 | 60.0 | 24.4 | 11.3 | 43.6 |
| Lemur-70B | 64.5 | 51.6 | 54.9 | 35.4 | 53.2 | 62.8 | 30.4 | 30.7 | 47.9 |
| Lemur-70B-Chat | 65.3 | 61.9 | 66.3 | 61.0 | 55.5 | 62.5 | 42.9 | 34.5 | 55.0 |

Reviewer comment (Evaluation):
As shown in Tables 4, 5, and 7, the performance of the proposed model Lemur-70B-Chat falls short when compared to GPT-4 and this discrepancy in performance lacks an explanatory or discussion.

Author response:
We appreciate your attention to the differences between GPT-4 and our Lemur model! To better address your concern about performance gaps and guide the development of future language agent models, we added a 3-part human error analysis section in Appendix D in the revised paper. We hope this section can provide a detailed analysis of the three scenarios you mentioned, and also reflect the underlying reasons for the discrepancy between GPT-4 and our model. We summarize them as the following:
- **Tool Usage ability:** We found that very complex and lengthy problems make it difficult for the models except GPT-4 to generate executable agent actions in the correct format. We found this problem when studying the M-MATH results. The long problems from MATH include solving equations described using LaTeX syntax, which is long and noisy. This misleads language models, which often generate unparsable responses or wrong actions. We hope to keep improving the ability of instruction following abilities in these scenarios. Here is part of our newly added Table 9:

| Model                        | Execution error | Invalid action |
|------------------------------|-----------------|----------------|
| Lemur-70B-Chat             | 29.00           | 26.00          |
| gpt-3.5-turbo                     | 17.00           | 15.00          |
| gpt-4                      | 7.00            | 2.00           |

- **Self-debugging ability:** Although the Lemur model is close to the performance of GPT-3.5-turbo in many environments of this scenario, we found that there is still a significant gap between Lemur and GPT-4 in challenging problems. We take InterCode-SQL as an example to study this issue. We categorize the problems of the SQL problems into different levels. According to the table, as the difficulty level increases, the performance gap between GPT-4 and Lemur gradually widens. This guides us to pay more attention to challenging cases. Here is part of our newly added Table 10:

| Model                        | Easy  | Medium | Hard  | Extra | All   |
|------------------------------|-------|--------|-------|-------|-------|
| Lemur-70B-Chat             | 92.74 | 74.89  | 70.69 | 45.78 | 73.79 |
| gpt-3.5-turbo                     | 92.74 | 74.89  | 67.24 | 43.37 | 72.82 |
| gpt-4                      | 95.16 | 82.96  | 86.21 | 68.67 | 84.14 |


- **Exploring in Partially-observable environment:** We manually researched the CTF (Capture The Flag) tasks, which is a popular competition program originating from cybersecurity. We manually labeled the problems in 100 CTF tasks and divided each problem into 6 categories. We found that GPT-4 and Lemur perform significantly better in the "general skills" category than in domain-specific fields such as "cryptography" and "reverse engineering". This means current language agents methods easily fail in domain-specific scenarios, which guide the future research of language agents. Please refer to our newly added Figure 5.

Please refer to Appendix D of our revised paper for more detailed information. We hope the added concrete detailed analysis can address your concerns!

Reviewer comment (Experiments):
Table 3 lists three baseline models—StarCoder-15B, StarCoderPlus-15B, and WizardCoder-15B—without corresponding explanations or references in the provided context.

Author response:
- We added references and explanations to baseline models StarCoder-15B, StarCoderPlus-15B, and WizardCoder-15B in Section 4.1 and Appendix C.
- In Section 4.5, we analyze the results shown in Table 7. In Appendix B, you may find the reference and introduction to the tasks InterCoder-CTF, WebArena, and ALFWorld. We also supplement the descriptions of baseline models in Appendix C.

Reviewer comment (Presentation):
Table 7 does not have references and analysis.

Author response:
- We added references and explanations to baseline models StarCoder-15B, StarCoderPlus-15B, and WizardCoder-15B in Section 4.1 and Appendix C.
- In Section 4.5, we analyze the results shown in Table 7. In Appendix B, you may find the reference and introduction to the tasks InterCoder-CTF, WebArena, and ALFWorld. We also supplement the descriptions of baseline models in Appendix C.

Reviewer comment (Novelty):
GPT4 also integrates text and code capabilities, what are the advantages of this paper?

Author response:
Thanks for your comments! We would like to discuss our value in two aspects: a research study on language agents with insights and open-sourced models.

- **Comprehensive language agent research study with insights:** Previous works usually studied language agents with specific tools and environments. In our work, we establish a comprehensive language agent study by organizing diverse environments to evaluate key agent skills in different scenarios. We are glad that this is recognized by all the reviewers.
- The insight into harmonious abilities motivated us to develop Lemur models, and our experiments indicate its direct importance to language agents compared with unbalanced models like Llama or CodeLlama. To provide more insights into language agent development,  we have added more human analysis in the appendix of this revised version, indicating the direction to further develop open language agents.
- **Open-sourcing effort value for community:** Current AI agent research largely relies on prompting close-sourced models like GPT-4, GPT-3.5 Turbo, and Claude-2. There are a lot of limitations when accessing close-sourced models by APIs, such as high cost and difficulties in custom training. We made expansive efforts to develop and open-source Lemur models. We believe that it will help the AI agent community develop and explore LLM-based agents by not only prompting models but also conducting agent training with different learning methods like RL. We are grateful to the Reviewer `QCzC` who also mentioned this point.
- In addition, we enjoy the natural advantages of open-source models, such as local deployment for privacy protection, transparent construction pipeline, and open-source community collaboration.

Reviewer comment (Novelty):
The technical contribution of this article is quite limited, it merely continues training the LLAMA model on a mixture of text and code data and instruction tuning on four datasets.

Author response:
Besides the technical contribution of demonstrating our training procedure, We would also like to highlight our research value and community benefits, including research study and insights to guide future language agent development, establishing comprehensive language agent evaluation, and open-sourcing models with superior performance for the language agent community. We are very grateful to `Ycpy`, `JHRb`, and `QCzC` for mentioning these contributions.

- **Research insights on language agent development:** Our insights into harmonious abilities motivated us to develop Lemur models, and our experiments indicate its direct importance to language agents compared to unbalanced models like Llama or CodeLlama. To provide more insights into language agent development,  We have further added more human analysis in the appendix of the revised version, indicating the direction to further develop open language agents.
- **Comprehensive language agent research study:** Previous works usually studied language agents with specific tools and environments. In our work, we establish a comprehensive language agent study by organizing diverse environments to evaluate key agent skills in different scenarios. We are glad that this is recognized by all reviewers.
- **Open-sourcing effort for language agents community as a key technical contribution:** Current AI agent research largely relies on prompting close-sourced models like GPT-4, GPT-3.5 Turbo, and Claude-2. There are a lot of limitations when accessing close-sourced models by APIs, such as high costs and difficulties in custom training. We made expansive efforts to develop and open-source Lemur models. We believe that it will help the AI agent community to develop and explore LLM-based agents by not only prompting models but also conducting agent training with different learning methods like RL. By being openly available, Lemur models can be accessed by individuals and organizations that might not have the resources to develop their own models from scratch, democratizing access to agent models.

Reviewer comment (Experiments):
When comparing performance on the code benchmark, the authors use a large 70B model, but the code-specific models they compare with are mostly 15-30B in size, which makes the comparison somewhat unequal.

Author response:
Thanks for your detailed feedback on the model comparison!
- To better track the performance of our model on code benchmark, we selected CodeLlama-34B and CodeLlama-34b-Instruct as representative baselines, which were the largest and most capable open-sourced code models.
- We would also like to highlight that our motivation for training the Lemur models is to harmonize the model’s natural language and code capabilities for agent tasks. Being the most capable model on code benchmarks may not be our focus. Instead, we may emphasize the model’s overall performance in both natural language, code, and agent tasks.

Reviewer comment (Experiments):
Why you use 10:1 text-to-code ratio in your pretraining data?

Author response:
- The ratio of code to text is a complex trade-off between the model size, the vanilla performance of the model, data quality, learning efficiency, catastrophic forgetting, etc.
- To roughly estimate a good ratio of composing the training corpora, we conducted a continue pre-training study with Llama models on a set of different code-text ratios and found that the ratio of 10:1 is an efficient ratio for the Llama model to transfer from text to text-code balance.
- Due to computational limits, it is difficult for us to conduct comprehensive experiments and perform systematic studies on this scale. However, we hope that our open-sourced effective settings and training checkpoints can benefit the community for continual exploration.
- We believe that a method to predict the optimal continue-pre-training data mixture ratio for a pair of domains to maximize their performance would be very meaningful and interesting, and it is still an open research question for the current large language model.

We have added this explanation in Appendix A of our revised version. Thanks for your valuable question!

Reviewer comment (Theory):
I am fully not convinced with the sentence "for the construction of language agents, it is imperative for language models to possess harmonized capabilities in both natural language and programming languages." It's unclear how programming languages correlate with language understanding. In fact, in the context of linguistics (morphology, syntax and semantics), programming languages might not satisfy any of them. I believe the authors should provide more context for it.

Author response:
**Response:** Thank you for your feedback regarding the motivation in our paper! We would like to further clarify our motivation:
- The most significant difference between agents and chatbots is that agents not only need to communicate with human users using natural language, but also need to interact with complex environments using executable programming languages. For example, the agent of web browsing accepts user instructions and controls the browser using code to complete tasks in multiple steps. 
- Therefore, this motivates us to directly enhance the code capabilities to help agents manipulate the environment via code. We would like to show the direct importance of programming language in agent scenarios, rather than the relevance or benefits of programming languages to natural languages.

Reviewer comment (Experiments):
Is there any reason to choosing scripting languages?

Author response:
**Response:**
- In section 2.1, we briefly mentioned that, among all languages, we focus on scripting or interpreted languages (Python, SQL, Bash, Perl, etc.) because agent models are often executed in interactive scenarios. Unlike compiled languages like C++, the interpreted nature of scripting languages allows immediate execution and easy modification, which is essential for dynamic interaction in language agent scenarios. 
- Another interesting finding is that, although the training corpora of Lemur contain a large proportion of scripting languages, we observed a general performance improvement on mainstream programming languages compared to the original LLaMa-2. The following are the detailed results of MultiPL-E in Table 3, which is a multi-programming language evaluation benchmark. 

| Model | C++ | Java | PHP | TypeScript | C# | Bash | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-2-70B | 30.40 | 31.70 | 34.20 | 15.10 | 25.90 | 8.90 | 24.4 |
| Lemur-70b-v1 | 34.78 | 35.40 | 35.40 | 35.40 | 25.47 | 15.82 | 30.4 |

Reviewer comment (Experiments):
Is the performance replicable for base models other than Llama?

Author response:
**Response:**
- Harmonizing model abilities by data mixture is a general and effective approach. For those base models other than Llama or smaller than 70B, we believe that it is generally capable of harmonizing natural language and code capabilities.
- When generalizing this approach to different models, the data mixture ratio and training steps need further adjustment. For example, smaller models have less capacity for harmonizing both text and code capabilities. Therefore, they may suffer from relatively obvious forgetting of natural language knowledge. The strategy of data mixture for large language model pretraining is an open and valuable research question. We believe we will have a more efficient and predictable data mixture strategy in future studies.

Reviewer comment (Experiments):
How much does the size of Llama matter for experiments? Can the same pipeline be replicated for smaller Llama versions?

Author response:
**Response:**
- Harmonizing model abilities by data mixture is a general and effective approach. For those base models other than Llama or smaller than 70B, we believe that it is generally capable of harmonizing natural language and code capabilities.
- When generalizing this approach to different models, the data mixture ratio and training steps need further adjustment. For example, smaller models have less capacity for harmonizing both text and code capabilities. Therefore, they may suffer from relatively obvious forgetting of natural language knowledge. The strategy of data mixture for large language model pretraining is an open and valuable research question. We believe we will have a more efficient and predictable data mixture strategy in future studies.