BOSE : A Systematic Evaluation Method Optimized for Base Models (2025)

Abstract: This paper poses two critical issues in evaluating base models (without post-training): (1) Unstable evaluation during training: in the early stages of pre-training, the models lack the capability to answer questions as required, leading to unstable evaluation results. This instability makes it difficult to provide solid conclusions to guide the training, especially for key experiments such as data ablation and scaling law. (2) Inconsistency between base and instruct models: base models generally exhibit poorer evaluation performance compared to corresponding instruct models. This gap poses a challenge for assessing whether a base model with better evaluation can truly lead to a better instruct model. To address these issues, we propose **B**ase model **O**riented **S**ystematic **E**valuation (**BOSE**), a method specifically designed to optimize the evaluation of base models. Specifically, BOSE introduces two key innovations: In-Context Light-instruction Prompt (**ICLiP**) for open-ended tasks and **Blank-ppl** for multi-choice tasks with candidate options, which transforms the standard perplexity (ppl) metric into a fill-in-the-blank format to mitigate early-stage evaluation fluctuations. Furthermore, we are the first to propose Kendall’s rank correlation to quantitatively measure the evaluation stability and consistency. Experimental results demonstrate that BOSE significantly enhances both the stability of evaluations during pre-training and the consistency between base and instruct models, thereby providing more reliable guidance for the LLMs’ training.

## introduction recently, large language models (llms) have demonstrated remarkable achievements across various domains @xcite . this has led to the development of numerous high-performing * equal contributions. llms @xcite @xcite . to evaluate the performance of these models on a wide range of tasks, an increasing number of benchmarks have been open-sourced @xcite . these benchmarks provide a comprehensive evaluation of the capabilities of llms, guiding the training and improving their weakness. evaluating large language models is a cornerstone in development of llms. generally, llms can be divided into two primary categories based on whether they receive post-training: base models and instruct models @xcite . instruct models undergo downstream adaptation, allowing them to adapt to specific tasks and answer questions as required. in contrast, base models focus on learning foundational knowledge without targeting specific tasks, resulting in a weaker ability to respond to instructions. consequently, most current benchmarks provide comprehensive evaluations of instruct models @xcite , while evaluations of base models are relatively rare. nevertheless, base models serve as the foundation for the entire llm training. an accurate evaluation for base models can provide essential insights for training llms, such as experiments on scaling laws @xcite , ablation of pre-training data, and the selection of appropriate base checkpoints for post-training @xcite . in this paper, we argue that the current evaluation is not suited to the characteristics of base models and lacks a systematic methodology. in a nutshell, the challenge arises from two dimensions: critical pre-training experiments, such as data ablation and scaling laws, rely on the performance of base models in the early stages of training. however, base models cannot follow instructions and solve problems during the initial stages, leading to fluctuations in evaluation results. figure 1 (left) illustrates the dynamic evolution of a 1b-parameter model's performance across different benchmarks as the training tokens increase. the significant fluctuations in the evaluation make it difficult to determine whether the model has gained real improvement throughout the training tokens. this instability hinders making reliable decisions for these critical experiments. in this paper, we propose a systematic approach for the evaluation of base models, called base model-oriented systematic evaluation (bose). bose aims to address the aforementioned critical issues by optimizing existing evaluation methods. specifically, we observe that base models often struggle to understand evaluation questions, which results in inaccurate reflections of their true capabilities. this is particularly evident in their difficulty with multi-choice questions, poor ability to follow complex instructions, and tendency to generate redundant continuations. bose improves evaluation techniques tailored to pre-training characteristics for two key tasks: open-ended generation and multi-choice. it introduces in-context lightinstruction prompt (iclip) to improve the base model's responses to open-ended questions and transforms multi-choice questions into fill-in-theblank versions (blank-ppl) to mitigate evaluation fluctuations in early stage. finally, we validate the effectiveness of bose by examining two key aspects: training stability and consistency between base and instruct models. this evaluation aims to determine whether the results exhibit a stable growth trend and whether a better base model can lead to a better instruction model. in summary, our contributions are as follows: to the best of our knowledge, this paper presents the first systematic framework for evaluating base llms, introducing empirically validated optimization methodologies with quantitative metrics. we believe this will yield actionable insights and practical recommendations to contribute to advance future base llm evaluation. ## related work in this section, we first introduce some commonly used benchmarks and evaluation tasks for llms, and then focus on the existing approaches for evaluating base models. evaluation tasks. based on whether there exist reference answers to be automatically calculated, evaluation tasks commonly can be classified as either ground truth-based evaluation or human preference-based evaluation @xcite . considering that base models typically cannot align with human preferences, we focus on evaluation tasks with ground truths. as for these benchmarks, the presence of candidate answers can generally further differentiate (open-compass, 2023): ## methodology different from instruct models, base model evaluation encounters unique challenges due to the lack of instruction-tuning and preference alignment. specifically, base models typically fail to comprehend the intent behind prompts or follow explicit instructions, leading to suboptimal evaluation protocols that neither reflect their intrinsic capabilities nor bring reliable evaluation results, which becomes particularly critical during early pre-training stages. to facilitate these limitations and align evaluation protocols with the intrinsic characteristics of base models, we optimize the evaluation methods for open-ended and multi-choice tasks respectively, as illustrated in figure 2 . ## experiments in this section, we describe systematic experiments to investigate how bose enhances evaluation stability during pre-training and ensures consistency between base and instruct models foot_0 . ## conclusion in this paper, we propose a systematic evaluation method tailored to base model, named bose, specifically designed to enhance the stability of base model evaluations during pre-training and ensure consistency with evaluations of instruct models. for open-ended tasks, bose develops a fewshot prompt template with light-instruction to guide the base model more effectively. for multi-choice tasks, bose innovatively transforms the standard perplexity (ppl) metric into a fill-in-the-blank format. these adaptations align better with the inherent characteristics of base models and facilitate more accurate evaluations to reflect the real capabilities of base models. moreover, we are the first to adopt kendall's rank correlation to quantitatively assess the stability and consistency of base models' evaluation. this metric provides a robust and reliable way to compare the evaluation results of the base models with those of corresponding instruct models. extensive experiments demonstrate the effectiveness and superiority of our proposed bose, validating its potential to significantly improve the evaluation and provide more meaningful insights into the true capabilities of base models.