Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints (2025)

Abstract: Evaluation of intermediate language model checkpoints during training is critical for effective model development and selection. How-ever, reliable evaluation using the popular multiple-choice question (MCQ) format is challenging, as small and non instruction-tunedmodels often lack the symbolic reasoning required for the task. This is despite the fact that MCQ evaluation is often used and needed todistinguish between the performance of different training runs. In particular, when prompted with a question and a set of labeled answerchoices (e.g., “A. . . . , B. . . . , C. . . . ”), many models struggle to emit the correct label (e.g., “C”), even when they can select the correct string answer choice. We propose an alternative evaluation method: fine-tuning the model on an auxiliary MCQ dataset prior to outputting labels. We validate this approach empirically by showing that training on auxiliary data improves MCQ ability on all our test datasets except 1. This approach provides a more accurate signal of model capability at intermediate checkpoints, as it disentangles the evaluation of core knowledge from the model’s emerging ability to follow formatting instructions.

## introduction robust and accurate evaluation of large language models (llms) is crucial for their development, guiding the design decisions model developers make when selecting from different model candidates. more specifically, it is common practice to evaluate intermediate model checkpoints over the course of a training run to estimate the final model's abilities before training is completed @xcite @xcite . therefore, it is important to have robust ways to evaluate these intermediate checkpoints. however, intermediate we propose fine-tuning on the mcq format prior to evaluation so that the fine-tuned model (right) learns to output the correct label ('b'). this allows for a more robust test of its underlying skills. we demonstrate this improves mcq evaluation reliably. one common lm evaluation format is that of multiple choice questions (mcqs; @xcite , which are easy to automatically score due to the existence of one pre-specified correct answer. in this format, the model is asked a question, given different answer choices, and must select the correct answer from the provided choices (figure 1 ). this format thus avoids the pitfalls of evaluating the correctness of open-ended llm-generated text. however, it often proves challenging in practice to evaluate intermediate model checkpoints on mcqs, because learning to answer mcqs is a skill in-and-of-itself that must also be learned during pretraining. more specifically, the ability to map a predicted answer choice string (e.g., "paris") to its respective symbol (e.g., "b") and then generate that symbol, known as "symbol binding" @xcite , is learned only after some number of pretraining steps @xcite . in light of this issue, how can we best standardize model evaluation across checkpoints of varying instruction-following abilities? prior work has proposed to evaluate each checkpoint with multiple formats and take the maximal score @xcite , but this approach both requires double the number of evaluations and adds complexity to results by introducing a format confounder. we investigate an alternative approach to evaluating intermediate model checkpoints on mcq datasets: fine-tune each checkpoint on an auxiliary mcq dataset (potentially from a different task) to teach the evaluation format, and then evaluate on the target dataset. this method gives the model explicit exposure to the multiple-choice format prior to evaluation, improving its ability to follow the format. this approach can thus give an arguably better estimate of a model's true capability on a given skill or domain, mitigating issues such as all answer choices being assigned low probability @xcite , answers differing based on evaluation format @xcite , or preambling @xcite . in this work, we address the following research questions: (1) can an intermediate model effectively acquire the ability to follow the mcq format through fine-tuning? (2) does this format learning on an auxiliary dataset transfer to improved performance on other, unseen mcq datasets? (3) how does the model's final evaluation accuracy scale with the number of auxiliary training examples? our empirical studies reveal three key findings. first, we demonstrate that intermediate models can effectively acquire the mcq format through auxiliary fine-tuning, and that this capability transfers across datasets. second, using a more diverse auxiliary dataset leads to stronger performance on the target task. finally, we find that model accuracy on the target dataset increases with the number of auxiliary training examples. taken together, these findings provide a practical methodology for more reliably evaluating and comparing intermediate language models on mcq tasks. ## current evaluation methodology for multiple choice questions there are two primary methodologies for evaluating models on mcq datasets: label-based and sequence-based formatting, with examples of each shown in figure 2 . in this context, the word "format" refers to both the prompt structure and the model's expected answer. label-based formatting assigns a symbol, such as a, b, c, or d, to each choice. the symbol with the highest probability is selected as the model's prediction. sequencebased formatting, by contrast, calculates which answer string the model is most likely to generate. the answer string with the highest probability is then selected as the model's prediction. ## difficulties with mcq evaluation evaluating language models on multiple-choice questions presents several challenges, with distinct problems arising from both of the primary evaluation formats. ## auxiliary format fine-tuning to address the challenges of standard mcq evaluation, we propose and investigate a two-stage methodology. first, an intermediate model checkpoint is briefly fine-tuned on an auxiliary mcq dataset. during this stage, the model is trained exclusively on the label-based format: given a question and choices mapped to symbols, it learns to output the single token for the correct answer. second, this newly fine-tuned model is evaluated on the target mcq dataset using the same label-based format. this approach is designed to disentangle a model's underlying knowledge from its ability to follow a specific format, thereby mitigating issues from both standard evaluation techniques. the finetuning stage explicitly teaches the skill of symbol binding, addressing the primary failure point for intermediate models in standard label-based evaluation. this process also targets format-specific artifacts; because the correct symbol's identity and position are varied across training examples, inherent label bias is reduced. similarly, training the model to maximize the first-token probability of the correct symbol inherently penalizes the generation of any preamble. crucially, our method retains the primary strengths of the original formats. after the onetime fine-tuning, evaluation remains computationally efficient, requiring only a single forward pass per question. by using label-based prediction, it also completely avoids the problem of surface form competition inherent to sequence-based evaluation. ## related work while mcqs are commonly used to evaluate llms due to their simplicity and efficiency @xcite , the reliability of these evaluation methods is disputed. prior work has identified many issues with mcq evaluation. for instance, there seem to be inconsistent results when comparing probability-based scoring (which encompasses both sequence-based and label-based formatting) and generation-based scoring @xcite . additionally, @xcite demonstrates that surface form competition can cause sequencebased formatting to underrepresent model ability significantly. many authors have also pointed out that option order has a large effect in label-based formatting @xcite @xcite . efforts to improve mcq robustness have focused on mitigating biases in scoring methods. for example, @xcite proposed addressing position bias by finding the prior probabilities that the llm would place on each position, while holtzman et al. (2021) addresses surface form competition by reweighting answer likelihoods. however, the efficacy of such methods remains inconsistent: wiegreffe et al. (2023) demonstrates that increasing probability mass on answer choices can paradoxically harm accuracy for certain llms. while some studies advocate for task-specific calibration @xcite , others caution against these methods of correcting for biases since they may not generalize across models or datasets @xcite . our method of fine-tuning a model to follow a specific format is conceptually related to instruc-tion tuning @xcite , where a pretrained model is further trained on a collection of instructions and desired responses. however, a key distinction lies in the goal and application. instruction tuning is typically a largescale, final training stage meant to create a generalpurpose, obedient model. in contrast, our method is a lightweight, targeted fine-tuning step designed specifically as a pre-evaluation probe to assess the knowledge of intermediate checkpoints. it is therefore a tool for evaluation rather than a final step in model creation. most similar to our work is @xcite , who also finetune intermediate model checkpoints and evaluate performance as a means to predict when and whether certain "emergent" skills will be learned, some of which are instantiated as mcqa datasets. however, their goal is not to predict the success of any particular training run or standardize evaluation format, but rather to predict scaling laws for emergent behaviors. ## conclusion in this paper, we address issues with evaluating intermediate llm checkpoints on mcq-style datasets. standard evaluation methods such as sequence-based and label-based formatting have significant issues that make them ill-suited candidates for evaluation. scoring with label-based formatting is impossible when the model does not have the capability to symbol bind, and sequencebased formatting suffers from surface form competition as well as numerous other issues. to mitigate these problems, we propose fine-tuning on an auxiliary mcq dataset followed by scoring with label-based formatting on the target datasets. this allows models to explicitly learn the mcq format while reducing bias and improving robustness. the empirical results we present in this paper demonstrate that this fine-tuning approach shows significant promise to improve evaluation consistency for intermediate model checkpoints. furthermore, we show that not much data is actually required to make significant improvements to labelbased formatted evaluation. we also demonstrate that this method provides a better metric to distinguish model ability in intermediate model checkpoints. we believe that this is a promising direction that requires further study.