“My Answer is C ”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models (2024)

Abstract: The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging. One common evaluation approach uses multiple-choice questions to limit the response space. The model is then evaluated by ranking the candidate answers by the log probability of the first token prediction. However, first-tokens may not consistently reflect the final response output, due to model’s diverse response styles such as starting with “Sure” or refusing to answer. Consequently, first-token evaluation is not indicative of model behaviour when interacting with users. But by how much? We evaluate how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. Our results show that the two approaches are severely misaligned on all dimensions , reaching mismatch rates over 60%. Models heavily fine-tuned on conversational or safety data are especially impacted. Crucially, models remain misaligned even when we increasingly constrain prompts, i.e., force them to start with an option letter or example template. Our findings i) underscore the importance of inspecting the text output as well and ii) caution against relying solely on first-token evaluation.

## introduction multiple choice questions (mcq) are one of the most popular evaluation formats for understanding the capabilities of large language models (llms), such as commonsense reasoning @xcite @xcite @xcite and truthfulness @xcite . they are also an important part of aggregated evaluation benchmarks such as mmlu @xcite , big-bench (bench authors, 2023) and general instruction: please read the multiple-choice question below carefully and select one of the listed options. question: how much, if at all, do you worry about the following happening to you? being the victim of a terrorist attack options: a. worry a little b. do not worry at all c. worry a lot d. refused answer: ## conclusion we compared first-token evaluation methods with the text output for multiple-choice questions and showed that the first-token evaluation heavily misrepresents the text output for instruction-tuned models. the results question the reliability of firsttoken evaluation for instruction-tuned language models, especially in settings where refusal is likely due to the sensitive nature of topics asked in the question. we also showed that the first-token evaluation is more sensitive to the prompt format and has more selection bias than text output. we suggest a more direct and realistic evaluation by directly inspecting the text answer to help better understand the llm's behaviour in real-life settings.