Paper: GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length (ICLR-2024)

Reviewer comment (Experiments):
The paper lacks comparison to the BERT [1] pre-training, which used a two-step pre-trianing approach with a growing context window length.

Author response:
The pre-training methods of BERT and GPT-NeoX are distinct. BERT employs a two-step approach with an expanding context window, focusing on masked language modeling and next sentence prediction. In contrast, GPT-NeoX utilizes an autoregressive training strategy. Considering these differences, we did not choose the BERT pre-training as a baseline.

Reviewer comment (Presentation):
The figures are confusing with different arrows pointing at the lines with same colors (Figure 3 & 5).

Author response:
Thanks for your suggestion, we modified the figure to make it clearer.

Reviewer comment (Writing):
Where are GrowLength-1 and GrowLength-2 defined?

Author response:
In Figure 4, 'GrowLength-1' represents the model trained using our growth strategy. In contrast, 'GrowLength-2' employs the same strategy but processes approximately twice as many tokens as 'GrowLength-1'. 'GrowLength-1' demonstrates accelerated training times compared to models using a fixed context length of 1024, as shown in Figure 2.

When processing more tokens, the training duration of 'GrowLength-2' is comparable to that of a fixed-length model, namely 'LLM1024'. We have included both models in the figure to ensure a comprehensive and fair comparison.

Reviewer comment (Experiments):
How would the GrowLength method work with substantially larger model 7B+?

Author response:
Larger models, such as those with 7B+ parameters, require significantly more computational resources. Implementing the GrowLength method at such scales could be prohibitively resource-intensive in terms of computational power and energy consumption. This limitation could affect the practicality of using the method in real-world applications, where resource constraints are a significant consideration. Moreover, it is common practice to use smaller models to verify experimental settings, as seen in the Llama2 paper. In this paper, the GQA setting is verified using a smaller model (30B) before being applied to a larger model (70B).

Also, it has been found that much smaller models' performance can predict the performance of huge models. This has been discussed in GPT-4 technical report[1], PaLM-2 report[2] and Chinchila[3]. For examples, by the figure 1 and figure 2 from GPT-4's report, much smaller models (1000x smaller) can be used to predict GPT-4's performance. Hence, we kindly argue that, the lack of results from pretty large models does not harm the soundness of our mehtods.

[1][GPT-4 Technical Report](https://arxiv.org/pdf/2303.08774.pdf)  
[2][PaLM 2 Technical Report](https://arxiv.org/pdf/2305.10403.pdf)  
[3][Training Compute-Optimal Large Language Models(Chinchilla)](https://arxiv.org/pdf/2203.15556.pdf)

Reviewer comment (Experiments):
How would the GrowLength method work with substantially context windows?

Author response:
**if we understand correctly, the reviewer want the performance of LLM with longer context windows**

The LLM pre-trained by our proposed method consistently demonstrates superiority over all the baselines when comparing GrowLength-1, LLM1024, and LLM128. The results are presented in Figure 4.

Reviewer comment (Experiments):
How to determine the growing context window size during training (like 128, 256, 512, …) is not rigorously studied. Will this exponential growth be too fast, especially for the latter stage, or even longer sequence (e.g., 1B tokens)? And maybe direct positional extrapolation will not work well when growing too fast.

Author response:
**Q1: How to determine the growing context window size during training (like 128, 256, 512, …) is not rigorously studies. Will this exponential growth be too fast, especially for the latter stage, or even longer sequence (e.g., 1B tokens)? And maybe direct positional extrapolation will not work well when growing too fast.**

Our growth schedule was determined empirically. We agree that significant discrepancies in stage lengths could impair performance during pretraining, as indicated in Figure 5. However, our experiments suggest that, at least in our current setting, exponential growth does not negatively affect our results. Regarding super-large context window sizes, the proposed methods may not be effective.

Some existing studies [1,2] have extended the context length by fine-tuning, for example, scaling Llama-2 from 4k to 8k or even 32k. This supports the viability of our method. Given the abundance of data in the pretraining phase compared to fine-tuning, we believe that a growth strategy could be effective even with larger length variations between stages.

It is also important to note that we do not claim our schedule is optimal. The key innovation of our research is the introduction of increasing context lengths during pretraining, which we believe enhances the process.

[1][Extending Context Window of Large Language Models via Positional Interpolation](https://arxiv.org/abs/2306.15595)  
[2][LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307)

Reviewer comment (Novelty):
The unique challenge when applying content window extension to pretraining stage is not very clear. It seems that simply using the technique proposed for fine-tuning also works well.

Author response:
**Q2:The unique challenge when applying content window extension to pretraining stage is not very clear. It seems that simply using the technique proposed for fine-tuning also works well.**

Our approach aims at reducing pretraining time for models with a fixed sequence length, such as 1024 tokens. While our method focuses on minimizing pretraining duration, we recognize that our fine-tuning technique is still applicable and beneficial for various purposes during the fine-tuning stage.

Reviewer comment (Evaluation):
No final testing results are presented. Although this paper is working on optimization but not generalization. But final performance on testing set of various tasks is the thing that matters a lot. Besides loss, these results should also be included and analyzed.

Author response:
**Q3: No final testing results are presented. Although this paper is working on optimization but not generalization. But final performance on testing set of various tasks is the thing that matters a lot. Besides loss, these results should also be included and analyzed.**

We conducted experiments to investigate the performance comparsion on other downstream tasks. Specifically, we ran several tasks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master). The results show that our proposed method outperforms the baseline in 7 out of 9 tasks.

| Dataset | Staged Training | Fixed Length Training(1024) |
|---------|---------|---------|
| boolq   | **0.4465**  | 0.3804  |
| multirc | **0.0189**  | 0.0105  |
| rte     | 0.5126  | **0.5235**  |
| wic     | **0.5235**  | 0.5047  |
| wnli    | **0.4507**  | 0.4366  |
| cb      | **0.5179**  | 0.4107  |
| qasper  | **0.3356**  | 0.0     |
| webqs   | **0.0009**  | 0.0005  |
| wsc     | 0.3654  | 0.3654  |

Reviewer comment (Experiments):
Will this method be sensitive to the length distribution of the training data? If the pre-training dataset has few samples with short sequences, will GrowLength still be effective?

Author response:
**Q4: Will this method be sensitive to the length distribution of the training data? If the pre-training dataset has few samples with short sequences, will GrowLength still be effective?**

Our GrowLength method remains effective even when the pre-training dataset contains a limited number of short sequences. Additionally, the nature of pre-training data for Large Language Models (LLMs) typically involves large volumes, allowing for the selective inclusion of pre-training text data. It is important to highlight that our proposed pre-training approach demonstrates superior generalization capabilities for longer texts, as illustrated in Figure 4. This indicates that our method is robust against variations in test text length.

Reviewer comment (Experiments):
What LLM is used in all experiments? I am curious about how this method works in experiments when applied to LLMs which already have some techniques for accelerating and handling long context, such as GQA in llama2 and FastAttention, given that the authors have explained that they are orthogonal, but experiments are more convincing here.

Author response:
**Q5: What LLM is used in all experiments? I am curious about how this method works in experiments when applied to LLMs which already have some techniques for accelerating and handling long context, such as GQA in llama2 and FastAttention, given that the authors have explained that they are orthogonal, but experiments are more convincing here.**

In our experiments, we employed the Pythia model([Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://arxiv.org/pdf/2304.01373.pdf)]. Our proposed training strategy is designed to be model-agnostic by altering only the input sequence length, which does not interfere with the core mechanisms of transformer models.

Specifically, the GQA technique in Llama2 operates at the architectural level and is independent of the input sequence length. Meanwhile, FastAttention functions at the computational level, serving as an equivalent to the traditional self-attention mechanism.

We acknowledge the value of further experimental validation. However, due to computational resource constraints, we were unable to include such results in our study.

Reviewer comment (Experiments):
My largest concern lies in the validation of the experiment. The model is trained around 20000s (5.56h) at most, which is too short to validate the pretraining process since the model is far away from convergence. Notice that [1] trains model for 300B tokens, and a 160M LLM usually needs several days to converge on 16 V100s, to my knowledge. I completely understand that resource demand is high for researchers; however, it is hard for me to agree with the conclusion from the present experiment.

Author response:
Thank you for your question and feedback. We conducted an experiment to continue pre-training the current models, as illustrated in Figure 1. As you pointed out, 'a 160M LLM usually requires several days to converge on 16 V100s,' we have decided to reserve the pretraining of larger models for future work. We are confident that our experiments, in their current form, have already substantiated that our proposed method can significantly reduce the time required for pretraining.

Reviewer comment (Novelty):
A shorter sequence surely leads to a smaller computation complexity. Therefore, this insight is not really original.

Author response:
Thank you for your observation. While it's true that shorter sequences reduce computational complexity, our experiment focused on the efficiency of training without compromising the model's quality. The key contribution of our work is not just identifying the reduction in computation due to shorter sequences, but also demonstrating how our specific methodology effectively maintains, or even enhances, the model's quality within these constraints. This aspect of our research extends beyond the straightforward observation of reduced computational complexity, offering a more nuanced and practical approach to model training.

Your feedback is invaluable to us, and we trust that our response has effectively addressed your concerns. In light of this, we would be grateful if the reviewer could reassess our work and possibly reconsider the score given.

Thanks,\
Authors

Reviewer comment (Experiments):
How do the authors transition between stages? Do the authors use position interpolation or extrapolation to provide a smooth transition when the sequence length increases? Furthermore, a comparison study to a different position embedding would highlight the importance of ROPE embeddings for a smooth transition across the stages.

Author response:
**Q(a) How do the authors transition between stages? Do the authors use position interpolation or extrapolation to provide a smooth transition when the sequence length increases? Furthermore, a comparison study to a different position embedding would highlight the importance of ROPE embeddings for a smooth transition across the stages.**

We thank you for the insightful and valuable question. Regarding the stage transition, through the experimental results in Figures 1, 2, and 3, we found that directly using position extrapolation makes the transition smooth enough.

For different position embeddings, we conducted the following experiments to investigate the impact of position embedding. These include various position embeddings, such as absolute positional embedding (GPT-2), and Alibi positional embedding (BLOOM).

Reviewer comment (Experiments):
Related to my second question, if the batch sizes have been changed at each stage, have the hyperparameters (Learning rate, batch size, etc.) of the baseline experiments been optimally tuned for fair comparisons?

Author response:
**Q(c) Related to my second question, if the batch sizes have been changed at each stage, have the hyperparameters (Learning rate, batch size, etc.) of the baseline experiments been optimally tuned for fair comparisons?**


Thank you for your question. We would like to explain the settings for hyperparameter tuning, such as learning rate and batch size. In large language model (LM) pretraining, it's common practice not to adjust hyperparameters, including learning rate, extensively. This is due to the unaffordable training costs. In line with this standard practice, we did not specifically tune these hyperparameters for each stage. However, we ensured that the comparisons remained as fair and consistent as possible within the constraints of our experimental setup.

Reviewer comment (Evaluation):
How does the improvement in perplexity relate to improvements in downstream performance? Any fine-tuned or zero-shot performance will show the general efficacy of the proposed method.

Author response:
**Q(d) How does the improvement in perplexity relate to improvements in downstream performance? Any fine-tuned or zero-shot performance will show the general efficacy of the proposed method.**

Thank you for your insightful comments regarding the relationship between perplexity improvements and downstream task performance. We appreciate the opportunity to further clarify and elaborate on this aspect of our work. You rightly point out the complexity in correlating perplexity improvements directly with downstream task performance. We agree that perplexity, while a valuable measure of a model's predictive capabilities, is not the sole indicator of performance in practical applications. We conducted experiments to investigate the performance comparsion on other downstream tasks. Specifically, we ran several tasks using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master). The results show that our proposed method outperforms the baseline in 7 out of 9 tasks.

| Dataset | Staged Training | Fixed Length Training(1024) |
|---------|---------|---------|
| boolq   | **0.4465**  | 0.3804  |
| multirc | **0.0189**  | 0.0105  |
| rte     | 0.5126  | **0.5235**  |
| wic     | **0.5235**  | 0.5047  |
| wnli    | **0.4507**  | 0.4366  |
| cb      | **0.5179**  | 0.4107  |
| qasper  | **0.3356**  | 0.0     |
| webqs   | **0.0009**  | 0.0005  |
| wsc     | 0.3654  | 0.3654  |


Your feedback is invaluable to us, and we trust that our response has effectively addressed your concerns. In light of this, we would be grateful if the reviewer could reassess our work and possibly reconsider the score given.

Thanks,\
Authors