Paper: Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
Their analysis of how pre-training loss correlates with gsm8k performance is pretty questionable because the loss numbers are for models trained on different datasets and thus are not comparable. While they admit this in the paper, it would be much better if they could make these numbers more comparable. One way to fix this would be to evaluate the loss of all the language models on some standard dataset.

Author response:
W1: We have calculated the pre-train losses of LLaMA, LLaMA2, and Pythia using the same pretraining dataset - The Pile test set and shown in Appendix H. We find that all our findings still hold.

Reviewer comment (Experiments):
They only consider gsm8k in this work. It would improve my confidence in their results if they also included other tasks, such as MATH.

Author response:
W2: We have added MATH in Appendix H.

Reviewer comment (Novelty):
Their RFT method is nearly a special case of STAR [cite], with the only difference being that they also filter reasoning paths for diversity. It therefore is unclear to me to what extent this could be claimed to be a new method.

Author response:
W3: We do not mean to propose a new data augmentation algorithm. We want to use RFT to state that augmented fine-tuning performance in math reasoning is based on different reasoning path amount instead of augmented data amount.

Reviewer comment (Experiments):
Some of the experiments are shown for up to 70B and others only up to 13B. Would it be possible to present results for all models sizes for all of the different scaling figures?

Author response:
Q1: We have conducted SFT, RFT-U13B, and RFT-U33B experiments for all models. We do not conduct RFT (k=100) for models larger than 33B and the reason is two-fold. The first reason is sampling 100 times with 65B/70B models is not cheap (10^19 FLOPs) and the second reason is we have shown that LLaMA-33B SFT models have shown a strong ability for over-fitting (or memorizing) GSM8K problems. LLaMA-33B SFT produces much less different reasoning paths compared to LLaMA-7/13B which is not suitable for RFT.

Reviewer comment (Novelty):
While the experiments are pretty convincing, the final conclusion does not seem to be very surprising or bring new insights. We already know the importance of model size, pre-training loss, and data amount. Since we are not going to design and pre-train a new LM at fine-tuning time, which requires predicting larger LM's performance from smaller ones, I'm not sure why a fine-tuning scaling law would be useful. It just suggests people use the strongest existing pre-training LM, which is exactly what people are doing right now. Maybe the authors can convince me of this during the rebuttal.

Author response:
To improve a specific ability for LLMs, we have three directions including (1) improve base model via pre-training (2) obtain more human-written fine-tuning data (3) augment more model-generated fine-tuning data. The fine-tuning scaling law (Section 3.2, (2)) and augmented fine-tuning scaling law (Section 3.3, (3)) can help LLM practitioners to decide to generate which kind of data (human-written fine-tuning data or model-generated data). When the amount of SFT data for a task (i.e. gsm8k) is not too much, one can decide to generate human-written data first. When the amount of SFT data is very large or fine-tuning scaling law is similar to augmented scaling law, one should decide to generate data from LLM augmentation.

Reviewer comment (Novelty):
The proposed RFT feels disconnected from the scaling law experiments. i.e. It does not seem to need to be inspired by the discovered scaling law. RFT basically suggests augmenting the fine-tuning data with LM-generated reasoning paths and rejecting the ones ending with wrong answers. I don't think the idea of data augmentation needs to be inspired by scaling law.

Author response:
RFT tries to investigate the augmented data fine-tuning scaling law. Our motivation is not to propose a new algorithm for data augmentation but to understand the augmentation data scaling law. The key finding of RFT is augmented fine-tuning performance in math reasoning is based on different reasoning path amount instead of augmented data amount.

Reviewer comment (Experiments):
The paper only does evaluation on 1 dataset, limiting its generalizability to other multi-hop reasoning problems.

Author response:
W1 & Q3: We have added Pythia on GSM8K and LLaMA on MATH experiments.

Reviewer comment (Evaluation):
Although the approach is intuitively simple, the amount of computational overhead added makes this approach not appealing.

Author response:
W2: In Appendix D, we have covered the RFT inference and training FLOPs, which have a significantly lower computational requirement (1 \times 10^{-4}) compared to pre-training. This level of computational cost is considered acceptable. Although some may argue that using GPT4 API for data augmentation is less expensive, this neglects the computational cost associated with inferencing using GPT4. Additionally, our approach solely relies on models like LLaMA that we aim to enhance, without the need for GPT4. In contrast, other math SFT papers like WizardMATH, Meta-Math, and Mammoth require GPT API for data augmentation.

Reviewer comment (Novelty):
The relationship between downstream task performance and pretraining loss has been well-established since the introduction of BERT. So the results in this paper is not really novel.

Author response:
W3: We are also interested in investigating the relationship between reasoning performance with human-written fine-tuning data amount and model-augmented fine-tuning data amount which is not be researched previously. We also find the distinct reasoning paths are the most important factor for augmentation dates.

Reviewer comment (Experiments):
Do the authors think the findings of this paper could be generalized and extended to other domains beyond mathematical reasoning tasks? Do the authors expect similar scaling relationships and performance improvements to hold in other LLM applications? Would be nice to see this supported with additional experiments.

Author response:
W1 & Q3: We have added Pythia on GSM8K and LLaMA on MATH experiments.

Reviewer comment (Experiments):
Can the authors show similar results with other models than just llama?

Author response:
General response: Many reviewers are interested in performance for different LLM and different datasets and question our conclusion because of the pre-training losses come from different corpora. We have uploaded a new version with new experiments. We use a random sample of the Pile test corpus to align the pre-train losses of LLaMA, LLaMA2, and Pythia series. We conduct SFT and RFT experiments on GSM8K benchmark with Pythia series. We conduct SFT experiments on MATH benchmark with LLaMA and LLaMA2. Our detailed results are listed in the updated PDF of section H in appendix. I will summary our main findings here.  

(a) The pre-training losses are still negatively linear correlated to SFT performance including Pythia series models.  
(b) RFT improves performances of Pythia series models significantly. (Pythia-410M SFT 5.6 vs RFT 18.9, Pythia-2.8B SFT 18.8 vs RFT 34.6).  
(c) For MATH benchmark, LLaMA series models still improving performance linearly when double fine-tuning data.