Paper: At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR-2024)

Reviewer comment (Evaluation):
Table 7 is confusing. The results in Table 7 show the code data will lead to a performance drop on four out of five datasets. It indicates that the code data may not help to improve the reasoning capability of LLMs. The authors should provide valid reasons.

Author response:
## **Table 7**
Our paper extensively evaluates various reasoning tasks, including logical and code reasoning, highlighting the positive impact of code-related data. Additionally, we sought to ascertain whether code data would affect common-sense tasks (which do not require high reasoning abilities). To address this, we conducted experiments outlined in Table 7, revealing that code data had minimal influence on performance across other tasks.

Reviewer comment (Novelty):
The related work is limited. Recently, there have been various papers discussing the reasoning capability of LLMs. Therefore, the authors should survey more related papers and compare with them.

Author response:
## **Related Work**
Thanks for your comment. We have added the following discussion to related work in our revised paper. This part has been highlighted in the paper. HELM test [2] provides a large-scale evaluation of existing large language models. They found that the code-cushman-001model trained on code data had more reasoning power than other natural language models. The AI2 [3] work also shows that, when equipped with complex thought chains, the code-davinci-002 model is currently the best-performing model on important mathematical benchmarks such as GSM8K. In addition, some work analyzed the reasoning ability of large language models through the way of thinking chain and found that because the original GPT-3 had not been subjected the code training, it could not do CoT reasoning [4]. The PaLM training data contains 5% of the code training data, which shows that PaLM can effectively perform CoT reasoning. However, in the above work, the evaluated models have different parameters and data sizes, and there are problems, such as unknown training details. Therefore, it is not rigorous to speculate the exact impact of code data on reasoning ability only by comparing existing models.

    [2] Liang P, Bommasani R, Lee T, et al. Holistic evaluation of language models[J]. arXiv preprint arXiv:2211.09110, 2022.
    
    [3] Fu Y, Peng H, Sabharwal A, et al. Complexity-based prompting for multi-step reasoning[J]. arXiv preprint arXiv:2210.00720, 2022.
    
    [4] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.

Reviewer comment (Writing):
Fix the grammar errors and improve the presentation. On page 8, “The experiment found that…’’ -> “The experiment showed that’’.

Author response:
## **Grammar Errors**
Thank you for your comments on the expression of the paper. We have corrected this grammar error in the revised paper. We will carefully correct the typos and further improve the presentation in the future.

Reviewer comment (Presentation):
Missing future work. The authors should provide the potential future work on LLMs based on the experimental results and insights provided in this paper.

Author response:
## **Future Work**
Models such as ChatGPT illustrate that larger-scale models will produce emergent capabilities. Therefore, the follow-up of this paper will study how to code the impact of data on models of different sizes and explore the relationship between code data and emergent capabilities. We have added these discussions to the revised paper and highlighted them.

Reviewer comment (Reproducibility):
Experimental details are insufficient. The paper may not provide enough details on experimental settings and parameter selection in some parts (such as data mixing strategies, decoding strategies, etc.). This might challenge researchers attempting to replicate or extend this work.

Author response:
## **Experimental Details**
+ **Data Resources**. The data for pre-training is collected from public datasets, including BaiDuQA, CAIL2018, SogouCA, and CodeParrot, and crawled data, including Common Crawl, Encyclopedias, News, and e-books. The constructed corpus contains about 42B tokes.

+ **Data Cleaning**. We use the rule-based data cleaning and model-based data filtering methods to ensure the quality of the data. More details can be found in Section F in the Appendix of the original paper.

+ **Data Mixing Strategy**. The mixing ratios of different data resources are demonstrated in Figure 2 of our original paper. Concretely, the ratios of CodeParrot, Common Crawl, e-Books, News, Encyclopedias, and Public Datasets are 36.1\%, 6.00\%, 13.50\%, 13.50\%, 14.00\%, and 16.90\%, respectively. Besides, to ensure that the model is not biased towards any specific data type during the pre-training process, we randomly shuffle the samples in the training corpus. In addition, during the instruction-tuning phase, we discuss the impact of different data mixing strategies on model performance. For details, please refer to Section 3.3.4 of the original article.

+ **Decoding Strategy**. For the generation task, we adopt a unified decoding strategy for all models, namely, the top-k sampling strategy, where k equals 3. Besides, for the classification task, we adopt a perplexity-based greedy decoding method. Concretely, the category with the minimum perplexity value is regarded as the predicted label of the paragraph.

+ **Parameters**. The detailed configurations, including environmental parameters, model parameters, optimization parameters, and parallelism parameters are summarized at Table 9 in the Appendix of the original paper.

Reviewer comment (Experiments):
Recently, various code foundation models, such as CodeLlama [1], have been opened. However, the authors do not conduct any discussions or experiments on them. In my opinion, the reasoning capability of code foundation models is also an essential part of the interest scope of this work.

Author response:
## **CodeLLama**
Thank you for sharing, we have noticed the release of Code Llama, a high-performance model focusing on the code field. However, this article focuses on how code data affects the reasoning capabilities of general LLM, while Code Llama focuses on improving the code capabilities of LLM. We carefully read the training process and result evaluation of Code Llama, and also discovered some interesting phenomena.

* **Code Reasoning Ability**. Code Llama continues to pre-train on code data based on Llama 2. We have observed the improvement in code ability brought about by the increase in code data.

* **Mathematical Ability**. We have observed from the open LLM Leaderboard that Code Llama' s training on more code data (non-mathematical data) has also improved mathematical ability. We infer that it may be related to the improvement of LLM's general reasoning ability by code data, or it may be related to the consistency between mathematical data and code data. We will add our discussions to related work, as shown in the figure below.

  | Model            | HumanEval (CODE) | GSM8K (MATH) |
  |------------------|------------------|--------------|
  | Llama 2-7B       | 12.2             | 3.49         |
  | Code Llama-7B    | **33.5**             | **5.16**         |
  | Llama 2-13B      | 20.1             | 10.84        |
  | Code Llama-13B   | **36.0**             | **12.13**        |

    *Note: HumanEval scores are from the original Code Llama report, GSM8K scores from the Open LLM Leaderboard(https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) .*

Reviewer comment (Novelty):
The related work part is weak. Missing important papers, such as [1,2,3].  
[1] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.  
[2] Yang A, Xiao B, Wang B, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint arXiv:2309.10305, 2023.  
[3] Li P, Sun T, Tang Q, et al. CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors[J]. arXiv preprint arXiv:2305.05711, 2023.

Author response:
## **Related Paper**
Thank you for providing relevant work references [4-6]. We have added these discussions to the revised paper and highlighted them. 
    
    [4] Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023. 

    [5] Yang A, Xiao B, Wang B, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint arXiv:2309.10305, 2023. 

    [6] Li P, Sun T, Tang Q, et al. CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors[J]. arXiv preprint arXiv:2305.05711, 2023.

Reviewer comment (Reproducibility):
Experimental details. In the paper, certain key sections, such as data mixing strategy and decoding strategy, do not seem to give sufficient details of experimental settings and parameter selection. This lack of information can lead to difficulties in reproducing and scaling. How was it set up by the author in the actual experiment?

Author response:
## **Experimental Details**
+ **Data Resources**. The data for pre-training is collected from public datasets, including BaiDuQA, CAIL2018, SogouCA, and CodeParrot, and crawled data, including Common Crawl, Encyclopedias, News, and e-books. The constructed corpus contains about 42B tokes.

+ **Data Cleaning**. We use the rule-based data cleaning and model-based data filtering methods to ensure the quality of the data. More details can be found in Section F in the Appendix of the original paper.

+ **Data Mixing Strategy**. The mixing ratios of different data resources are demonstrated in Figure 2 of our original paper. Concretely, the ratios of CodeParrot, Common Crawl, e-Books, News, Encyclopedias, and Public Datasets are 36.1\%, 6.00\%, 13.50\%, 13.50\%, 14.00\%, and 16.90\%, respectively. Besides, to ensure that the model is not biased towards any specific data type during the pre-training process, we randomly shuffle the samples in the training corpus. In addition, during the instruction-tuning phase, we discuss the impact of different data mixing strategies on model performance. For details, please refer to Section 3.3.4 of the original article.

+ **Decoding Strategy**. For the generation task, we adopt a unified decoding strategy for all models, namely, the top-k sampling strategy, where k equals 3. Besides, for the classification task, we adopt a perplexity-based greedy decoding method. Concretely, the category with the minimum perplexity value is regarded as the predicted label of the paragraph.

+ **Parameters**. The detailed configurations, including environmental parameters, model parameters, optimization parameters, and parallelism parameters are summarized at Table 9 in the Appendix of the original paper.

Reviewer comment (Experiments):
**Datasets:** I was not familiar with some of the datasets used in this work and after looking into some of them, I could not get a sense of how general and challenging they are. The majority of the computation cost for this project seems to be on the training stage, so I believe reporting results on a few more datasets (maybe only for Tables 2 and 3) can strengthen the main arguments of the paper. That could include datasets from other reasoning domains (e.g., math might be an important one that does not appear in the results) or from the same domains but on datasets that are more established.

Author response:
## **Datasets**
In terms of performance evaluation, the datasets used in this paper are all public datasets, which are widely adopted by many methods for testing model performance. For example, the Logic data comes from C_Eval [1], and models such as ChatGLM [2] and Qwen [3] have been tested on this dataset. Besides, MBPP [4] is a widely used code testing benchmark and is widely used by models such as GPT-4 [5], Llama 2 [6], etc.

    [1] Huang Y, Bai Y, Zhu Z, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models[J]. arXiv preprint arXiv:2305.08322, 2023.

    [2] Zeng A, Liu X, Du Z, et al. Glm-130b: An open bilingual pre-trained model[J]. arXiv preprint arXiv:2210.02414, 2022.

    [3] Bai J, Bai S, Chu Y, et al. Qwen technical report[J]. arXiv preprint arXiv:2309.16609, 2023.

    [4] Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint arXiv:2108.07732, 2021.

    [5] https://openai.com/research/gpt-4

    [6] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.


+ **General and Challenging Datasets**. 

  + **For the general test**, to comprehensively test the reasoning abilities of the models, we used different types of reasoning tasks and datasets for performance testing, including logical reasoning, legal reasoning, scientific question and answer, and code generation. These tasks comprehensively reflect the model's performance on different types of reasoning capabilities.

  + **For the challenge**, the evaluated models need to achieve promising reasoning results on all types of reasoning tasks, which is very challenging. Besides, the datasets used in this article usually require multi-step reasoning to obtain correct results, further improving the difficulty of reasoning.

Reviewer comment (Experiments):
**Mixture experiment:** The experiment on exploring ways to mix code and text data is interesting, but given that adding code at the instruction-tuning stage was already shown to be not that effective, I wonder why it was tested for the instruction-tuning stage. I understand the high computation cost of pre-training, but it seems to me that given the previous set of results, this experiment makes sense mostly at the pre-training stage where we have seen that code data can be effective.

Author response:
## **Mixture Experiment**
As you said, due to resource constraints, we only conducted data mixing experiments during the instruction-tuning phase. In this experiment, we found that using a higher proportion of code data in the early stage can improve the specific code reasoning ability of the language model. We think this may have implications for some vertical scenarios that require fine-tuning to implement. At the same time, in view of the findings in the fine-tuning phase, we are also planning to explore data mixing experiments in the pre-training phase. We believe this is a promising direction. Thanks for your suggestion.

Reviewer comment (Evaluation):
**Language Inconsistency:** For the results in Section 3.3.5, the authors conclude that training with code data *has little negative impact* on the performance of other tasks. But the numbers in Table 7 don't seem to show little negative impact. The large impact on the DuReader dataset has been already pointed out by the authors. Moreover, on CMNLI the performance decreases from 45.07 to 43.49 which is almost equal in magnitude to some of the gains reported in Table 2. I believe the language should be more consistent on the amount of improvement/decrement that can be considered a significant amount for the datasets and the experimental setup of the paper.

Author response:
## **Language Inconsistency**
Thanks for your suggestion. After carefully verifying the results of our article, particularly in section 3.3.5, we found that the code data had an impact on the DuReader task. We believe this may be due to the model not fully learning from the code and text data, leading to confusion when generating answers to reading comprehension questions. To avoid misleading the readers, we have improved our conclusion in the revised paper to state: "Using code data for training may negatively impact the performance of other tasks."

Reviewer comment (Presentation):
**Minor suggestions:** 1- In Table 3, there are multiple equal numbers but only one of them is in bold face. 2- In Table 4, I suggest reversing the rows and columns to make it consistent with the other tables.

Author response:
## **Minor Suggestions**
Thanks for your suggestions.
- Regarding the bolding of identical performances, we have made the corrections in the revised paper. 
- For Table 4, following your suggestion, we have reversed the rows and columns to make it consistent with other tables.

Reviewer comment (Experiments):
For the Logic dataset, I see that the accuracy for many different models is 40.9. This value appears in Table 2, Table 3 and Table 6. Does this hint at a potential problem in this dataset?

Author response:
+ **Logic Dateset**.
Thanks for your reminder. We carefully inspected the Logic dataset and the output of models. We found that the distribution of the predicted scores is different between different models. Besides, some of the answers to the questions overlapped while others were different. The similar low accuracy might be due to the difficulty of the Logic dataset.

Reviewer comment (Experiments):
Some of the datasets don't seem to be available in English (or at least I could not find an English version of them). Could you include some examples from these datasets translated to English, so the reader can get a sense of the nature and the difficulty level?

Author response:
+ **Examples of Datasets**.
Sorry for causing trouble for your understanding. For the non-English data sets used in this article, following your suggestion, we have translated them into English for display. Below are examples from the Logic, JEC-QA, and E-KAR datasets. We have added these examples to the revised paper and highlighted them. 

  + **One Example of Logic (Logical Reasoning)**
    
        Problem: 

        Regarding the physical education standard test for Class A, three teachers made the following predictions: Teacher Zhang said, "Not everyone will fail." Teacher Li said, "Someone will fail." Teacher Wang said, "Both the class president and the study committee member will pass." If only one of these teachers' predictions is correct, which of the following must be true?

        Answer List:

        A: "Both the class president and the study committee member failed."
        B: "Both the class president and the study committee member passed."
        C: "The class president passed, but the study committee member failed."
        D: "The class president failed, but the study committee member passed."

        Answer: A

  + **One Example of JEC-QA (Legal QA)**


        Problem: 

        A miscellaneous article written by person A caused a significant stir after its publication. The article was reprinted by several newspapers and websites without compensation. Person B translated the article into French and person C translated it into Uighur. Both translations were published domestically without A's consent and without any remuneration. Which of the following viewpoints is correct?

        Answer List:

        A: "The act of newspapers and websites reprinting the article does not constitute infringement."
        B: "The actions of both B and C do not constitute infringement."
        C: "B's action does not constitute infringement, but C's action does."
        D: "B's action constitutes infringement, but C's action does not."
        
        Answer: C



  + **One Example of E-KAR (Analogical Reasoning)**

        Problem: 

        Based on the given relationship [Speed:Time:Distance], choose the option that fits this relationship.
        
        Answer List:

        A: "Interest Rate:Principal:Interest"
        B: "Quality:Variety:Quantity"
        C: "Profit:Cost:Value"
        D: "Income:Expenditure:Surplus"

        Answer: A 

        Explanation: From the given relationship, we infer the following: "Speed" multiplied by "Time" equals "Distance". In option A, "Interest Rate" multiplied by "Principal" equals "Interest". In option B, there is no clear logic connecting "Quality", "Variety", and "Quantity". In option C, the product of "Profit" and "Cost" is not "Value". In option D, "Surplus" is the difference between "Income" and "Expenditure". Therefore, the correct choice is A.

Reviewer comment (Experiments):
From my understanding (and correct me If I'm wrong), the code model is trained on more overall tokens than the NL model. I would expect a study like that to control for the number of pre-training tokens while changing their nature i.e., text vs. code. If the code model is trained on as many natural text tokens as the baseline model in addition to having code in the pre-training data, then the code model should be expected to perform better because it was trained on more data. No surprise there.

Author response:
## **Training Data**
Thanks for your insightful and constructive comments.

+ **Training Corpus**. Your understanding is partly correct. Actually, NL (2.6B) is trained on about 100G natural language data. Besides, CODE (2.6B) is trained on about 100G natural language data plus 50G code data. In addition, NL (13B) is trained on more than 150G natural language data. 

+ **Effectiveness**. In the pre-training and instruction tuning stages, we compare the above three models, including NL(2.6B), NL(13B), and CODE(2.6B). In the pre-training stage, Table 2 shows that, on the reasoning task, CODE (2.6B) with mixed code data and natural language not only outperforms NL (2.6B) with natural language but also outperforms the larger pure text model NL (13B) trained on more text data. These experimental results highlight the importance of code data in the pre-training stage. Besides, at the instruction tuning stage, Table 3 demonstrates that CC (2.6B), which is fine-tuned on code instructions, performs better on specific code tasks, highlighting the importance of code data in the instruction tuning stage.

+ **Additional Study**. Thanks for your insightful suggestion! We have already begun our pre-training of NL(2.6) on 150G natural language data. Due to the limitation of computational resources, it will take about 15 days. We will add the results and conclusions in the revised paper to further improve the comprehensiveness and quality of this paper. 

+ **More Data**. Actually, as claimed in **Effectiveness**, the fact that CODE (2.6B) beats NL (13B) on reasoning can show the importance of code data in improving the reasoning capacity of models. Besides, we do not agree that simply adding more data can definitely result in better performance. The reasons are as follows. 1) Large amounts of data may easily lead to underfitting problems of the models. 2) The low-quality data may lead to the hallucination problem in LLMs. 3) According to many scaling laws [1, 2, 3], the model size, dataset size, and the amount of training computation should be amplified simultaneously to achieve better performance. Therefore, according to the extensive experiments and analyses as claimed in **Effectiveness**, we demonstrate the importance of the code data in both pre-training and fine-tuning of LLMs. 

      [1] Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint arXiv:2001.08361, 2020.

      [2] Hoffmann J, Borgeaud S, Mensch A, et al. Training compute-optimal large language models[J]. arXiv preprint arXiv:2203.15556, 2022.

      [3] OpenAI R. Gpt-4 technical report. arxiv 2303.08774[J]. View in Article, 2023, 2.

Reviewer comment (Evaluation):
It's hard to say whether the results reported have statistical significance. For example, in Table 2, the code model is only 0.13 points better than the NL model on ScienceQA. Are these results significant? And are these enough to conclude that code reasoning? I would expect the authors to run a statistical significance test to support their results.

Author response:
## **t-Test**
Thanks. 
+ In order to comprehensively evaluate the model's reasoning ability, we used different types of reasoning tasks and datasets for testing, including logical reasoning, legal reasoning, scientific question and answer, and code generation. These tasks comprehensively reflect the model's performance on different types of reasoning capabilities. Due to the diversity and complexity of these reasoning tasks, achieving promising performance on all tasks is a challenge. In Table 2, compared with NL (2.6B), we find that CODE (2.6B) can bring 4.54% and 3.88% performance improvement on MBPP and E-KAR datasets. 

+ Besides, following your suggestion, in order to illustrate the significance of these results, we conducted a t-test on the predicted scores in the following table. It demonstrates that all p-values are less than 0.05, indicating that the results are statistically significant. We have already add these results to the revised paper and highlighted them.

  | Dataset | p-value (<0.05) |
  |-----------|-----------------|
  | Logic     | 4.197e-06       |
  | JEC-QA    | 1.956e-25       |
  | ScienceQA | 0.014          |
  | E-KAR     | 7.013e-07       |
  | CosQA     | 1.066e-40       |

Reviewer comment (Evaluation):
The proposed dynamic mixing strategy produces very marginal improvements (except over logical reasoning) and has a negative effect on the performance over three tasks. The paper does not thoroughly investigate why this is the case. Also, the design of the mixing strategy seems rather arbitrary.

Author response:
## **Dynamic Mixing Strategy**
Thanks for your concern, which can help us to improve the paper. 

- In order to comprehensively evaluate the model's reasoning ability, we used different types of reasoning tasks and datasets for testing, including logical reasoning, legal reasoning, scientific question and answer, and code generation. These tasks comprehensively reflect the model's performance on different types of reasoning capabilities. Due to the diversity and complexity of these reasoning tasks, achieving promising performance on all tasks is a challenge. We admit that the dynamic mixing strategy can not achieve the best results on all reasoning tasks. But it is really an interesting idea that can improve the reasoning capacity of LLMs during training. We have not carefully studied the segment setting and mixing ratios. The experimental setting is a preliminary attempt. Therefore, further improvements can be made in the future, such as setting up more segments and analyzing different ratios. We believe this research question will be a promising direction. 

- Still, under these settings, we obtained some interesting and insightful experimental conclusions in the original paper, especially on the coding task. We have added these details in the revised paper and highlighted them.
  - Using a larger code proportion in the early stage can improve the performance of LLM in coding tasks (CosQA and MBPP). The reason may be that a higher code proportion in the early stage can better activate code-related reasoning capabilities under a higher learning rate.
  - The descending strategy can improve the performance of logic. Since the code data is more logical, giving more codes in the initial stage may improve the performance of logic reasoning.
  - In the other three datasets, uniform is better, probably because these tasks require both logical reasoning ability and common sense as well as natural language understanding ability.
 Therefore, we recommend choosing different hybrid strategies based on the characteristics of different downstream tasks.

- Designs
  - The experimental settings of the proposed dynamic mixing strategy are a preliminary exploration. We divided the experiment into four stages based on experience and the proportion of the dataset. Based on your suggestions, we will carefully explore more training stages in the future. We believe it will be a promising and useful training strategy. 
  - In order to maintain the ability of the general model, we do not set widely different stage proportions. Because a ratio that is too high or too low may cause the model to turn into a proprietary code model or a text model. Among them, the 5:3 ratio in the uniform sampling stage is the original text and code instruction data ratio.

Reviewer comment (Evaluation):
The evaluation does not cover mathematical reasoning, although it's one type of reasoning where we should expect great improvements since code data is roughly similar to math data.

Author response:
## **Mathematical Reasoning**

Following your suggestion, we selected the high school mathematics and high school physics problem parts of the MMLU [6] test set to evaluate the model in the pre-training stage. MMLU is a currently widely used dataset to evaluate the comprehensive ability of LLMs [7, 8], among which mathematics and physics can better reflect the reasoning ability of the model. The results obtained and the conclusions are shown in the table below. We have added these results to the revised paper and highlighted them. Thanks for your suggestions again.

    [6] Hendrycks D, Burns C, Basart S, et al. Measuring massive multitask language understanding[J]. arXiv preprint arXiv:2009.03300, 2020.

    [7] https://openai.com/research/gpt-4

    [8] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.



  | Task       | NL (2.6B) | NL (13B) | **CODE (2.6B)** | p value| *LLaMA (7B)*  |
  |------------|--------|-------|--------------|------------------------------------------|---------|
  | MMLU_Math  | 24.16*  | 22.30 | **24.91**   | <0.05    |  *24.97*  |
  | MMLU_Physics| 20.00 | 22.67* | **26.67**    | <0.05    |  *27.97*  |

  *Note: Source of LLaMA results: https://github.com/baichuan-inc/Baichuan-7B*

  In this table, **bold** values denote the best result, and the values* denote the runner-ups in the scope of this paper. The t-test is conducted to verify the significant improvement of the best values compared with the runner-up values. Besides, the italics denotes the models out of the scope of this paper. From these experimental results, we have three conclusions as follows. 

  + On mathematical and physical reasoning tasks, the CODE (2.6B) model shows advantages over NL (2.6B) and NL (13B), which strengthens the effectiveness of introducing code data in the pre-training stage. We performed a t-test for statistical significance, and the results showed that the p-value was less than 0.05, which indicated that the results were statistically significant.
  + We admit that models, including LLaMA (7B) and CODE (2.6B), perform relatively poorly on mathematical and physical tasks, but we believe that the relative improvement brought by code data is trustworthy. This also shows how it is necessary to further improve mathematics-related tasks.

Reviewer comment (Experiments):
The choice of the baseline model is unclear. Why use PanGu2.6B as your baseline model?

Author response:
## **Model Choice**
- PanGu, along with other LLMs like Llama and Google PaLM, shares a common architecture based on the GPT-2 decoder-only architecture and next token prediction task. In addition, due to resource limitations, we selected the 2.6B model for experiments and compared it with the 13B model. Because open-source models such as CPM2.6B [3], EVA2.6B [4], and ChatGLM2.6B [5] all use the 2.6B scale and show that they already have some general capabilities.

- At the same time, we look forward to the emergence of LLMs with different architectures than GPT, and we also look forward to following up and verifying more LLMs with different architectures.

- In this research, the GPU and funding supporters adopt the PanGu model. Similar facts are that Google prefers PaLM, OpenAI prefers GPT, and Meta prefers LLaMA. We focus more on exploring the research question itself, i.e., at which stage can code data help LLMs reasoning. The model choice should not be an advantage or a disadvantage. 

      [3] Zhang Z, Han X, Zhou H, et al. CPM: A large-scale generative Chinese pre-trained language model[J]. AI Open, 2021, 2: 93-99.

      [4] Zhou H, Ke P, Zhang Z, et al. Eva: An open-domain chinese dialogue system with large-scale generative pre-training[J]. arXiv preprint arXiv:2108.01547, 2021.

      [5] https://github.com/THUDM/ChatGLM2-6B