Metric assessment protocol in the context of answer fluctuation on MCQ tasks (2025)

Abstract: Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently. A variety of metrics can be employed for this task. However, previous research has not conducted a thorough assessment of them. At the same time, MCQ evaluation suffers from answer fluctuation: models produce different results given slight changes in prompts. We suggest a metric assessment protocol in which evaluation methodologies are analyzed through their connection with fluctuation rates, as well as original performance. Our results show that there is a strong link between existing metrics and the answer changing, even when computed without any additional prompt variants. Highest association on the protocol is demonstrated by a novel metric, worst accuracy.

## introduction testing on question answering tasks has become standard in the llm evaluation field @xcite . however, assessing models' generations in these conditions is a complex task, due to inapplicability of "traditional" metrics, such as bleu @xcite , rouge @xcite , or bertscore @xcite , because of high variation between possible correct answers @xcite . while human evaluation can be used instead, it can be costly @xcite and subjective @xcite . thus, multiplechoice questions (mcq) benchmarks have prevailed in llm evaluation, as a tool that maps all possible responses to a small set of options, with examples such as arc @xcite , gpqa @xcite , and bigbench-hard @xcite . using mcq tasks allows for the exact matching of answers selected by models and correct ones and for the computation of standard metrics, such as accuracy @xcite @xcite . while reporting accuracy is typical, the metrics available for mcq tasks include other possibilities. for instance, continuous metrics such as probability mass of correct answer can improve signal-to-noise ratio in evaluations @xcite or better track actual performance of models of different sizes during training @xcite . additionally, new metrics were proposed specifically in the context of mcq evaluation (e.g. @xcite . however, previous work has not provided a thorough comparative analysis of these metrics. in addition, prior research @xcite @xcite @xcite indicates that llms are sensitive to changes in mcq options order: it is possible to elicit a different response from a model simply by rearranging the proposed answers. the phenomenon of llms producing different answers given semantically insignificant prompt changes can be called answer fluctuation @xcite or answer floating @xcite . a deep understanding of answer fluctuation is crucial since llms' reliability remains a concern, especially in sensitive domains @xcite @xcite . nevertheless, discovering all cases of fluctuation leads to significantly higher computation costs, due to the necessity of testing multiple prompts. we propose to use this factor in order to compare metrics available for the evaluation of mcq tasks. in particular, we perform the costly calculation of models' responses fluctuation on all possible permutations and then compare those results with metrics computed on smaller subsets of permutations, assessing if any of the metrics could be used as a cost-efficient proxy for the full fluctuation rates (computed on all permutations), without losing the information about the original performance. our contributions can be summarized as follows: we find that most metrics strongly correlate with the full fluctuation rates, even when calculated only on the original version of the benchmark. however, the correlation becomes stronger when adding results from multiple permutations, achieving the coefficient of determination r 2 > 0.9 for partial fluctuation rates (computed on subsets of permutations) and the novel metric, worst accuracy. ## context & related work mcqs have been widespread in the education field @xcite . they are characterized by presenting several answer options within a question body, typically accompanied by labels (e.g. a/b/c/d), where a correct answer can be one, several, or no labels. in the context of llms evaluation, however, mcq benchmarks come with a single correct label, see an example in figure 1 . the unique correct answer allows for comparing models' responses to it and obtaining accuracy. as for the extraction of a model's responses, one can compare probabilities of the next token given a question prompt and choose the most probable one as the model's selected label. another method prominent in the field, though not covered in this paper, is to allow models to generate an answer of arbitrary length and later classify it as one of the labels @xcite . previous research demonstrates that one can cause answer fluctuation by permuting questions, their options and/or labels. @xcite show that even minimal prompt paraphrases, e.g., replacing "have" with "include" in the question, impact models' performance. @xcite which of these will form new soil the fastest? ## metrics survey given the variety of metrics available for mcq evaluation, it is essential to provide a coherent formalization for each of them. this section presents our notation and permutation types used for computation. furthermore, we provide formulas for existing metrics. finally, we introduce a novel metric, that we call worst accuracy. ## assessment protocol having presented all the metrics, one can choose a multitude of assessment protocols. since computing all permutations and finding the full fluctuation rates is a costly venture, we argue that an appropriate metric for mcq evaluation would be highly representative of the full fluctuation rates computed in a lower-cost environment. therefore, we propose evaluating the correlation of the proposed methodologies with full fluctuation rates. however, a metric should still be illustrative of the model's accuracy on the original option order, since this represents the result of a model on a version it was exposed to. thus, we additionally propose the following protocol, illustrated in figure 2 : ## results this section presents the results of steps 4-6 of the protocol introduced above. to begin with, we compute the correlation of the metrics with full fluctuation rates using original order and permutation subsets. second, we compare the results when adding correlation with the original accuracy. lastly, we assess the impact of picking random permutations for metric calculation foot_8 . ## limitations & future work selection of permutations as demonstrated in the results, multiple metrics appear sensitive to the permutations chosen to compute them. while we observe this phenomenon, further study is required on the optimal approaches to permutation selection. other permutation types while we illustrated how strongly metrics correlate with fluctuation, we only considered option order permutations. as discussed in section 2, fluctuation can occur with question paraphrasing, changing option typography, replacing option labels, etc. further work needs to include these types of permutations in the assessment. ## conclusion in this paper, we presented a new protocol for metric comparison in the context of answer fluctuation that llms exhibit when options of mcq tasks are permuted. to achieve this, we reviewed, formalized, and computed existing metrics applicable to such benchmarks, and introduced a new metric, worst accuracy. when applying the evaluation framework, we discovered that: