Instability in Downstream Task Performance During LLM Pretraining (2025)

Abstract: When training large language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score.However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model.In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora.We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels.To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance volatility.We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.

## introduction large language models (llms) are typically developed through pretraining on large-scale corpora, during which the performance on downstream tasks evolves dynamically. understanding these training dynamics is essential for diagnosing issues during model development and ensuring the reliability, reproducibility, and effectiveness of the resulting models on downstream tasks. in particular, the stability of evaluation metrics plays a key role in model development: when evaluation metrics fluctuate erratically, they undermine the reliability of the evaluation and hamper fair comparison across checkpoints, models, or experimental settings. such instability may arise not only in evaluation signals based on next-token prediction, such as training loss, but also in downstream task metrics that are central to model evaluation. ## mitigation of instability through post-processing in the previous section, we observed that downstream task performance often fluctuates during the pretraining of large language models (llms), both at the task and example levels. these fluctuations hinder reliable evaluation and model selection, especially when performance at the final training steps is used to benchmark models. to address this challenge, we investigate post-processing methods that stabilize performance by integrating information from multiple adjacent checkpoints. in the following, we describe two techniques, checkpoint averaging and checkpoint ensemble, provide a theoretical motivation for their use, and empirically evaluate their effectiveness in mitigating instability at both the task and example levels. ## conclusion this study analyzed the instability of downstream task performance during the pretraining of large language models, focusing on performance fluctuations across checkpoints. we found that such instability occurs widely across tasks and model sizes. these findings suggest that downstream performance can fluctuate significantly even in the later stages of training, which can undermine the reliability and reproducibility of evaluation results. to address this, we investigated post-processing approaches that integrate multiple checkpoints. both checkpoint averaging and ensemble methods effectively reduced performance fluctuations at both the task and example levels, without modifying the training procedure. while checkpoint averaging contributed more to improving average scores, checkpoint ensemble was more effective in stabilizing example-level predictions. given its lower inference cost, checkpoint averaging is particularly well-suited for practical deployment.