Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning (2024)

Abstract: Questions involving commonsense reasoning about everyday situations often admit many possible or plausible answers. In contrast, multiple-choice question (MCQ) benchmarks for commonsense reasoning require a hard selection of a single correct answer, which, in principle, should represent the most plausible answer choice. On 250 MCQ items sampled from two commonsense reasoning benchmarks, we collect 5,000 independent plausibility judgments on answer choices. We find that for over 20% of the sampled MCQS, the answer choice rated most plausible does not match the benchmark gold answers; upon manual inspection, we confirm that this subset exhibits higher rates of problems like ambiguity or semantic mismatch between question and answer choices. Experiments with LLMs reveal low accuracy
and high variation in performance on the subset, suggesting our plausibility criterion may be helpful in identifying more reliable benchmark items for commonsense evaluation.

## introduction commonsense reasoning about everyday situations involves soft judgments about the relative plausibility or likelihood of different possible outcomes. if a wine glass falls, a very likely outcome is that it breaks, but another technically possible outcome is that it bounces (e.g., because it lands on a trampoline). datasets like the choice of plausible alternatives (copa; @xcite or ordinal common-sense inference @xcite highlight this graded nature of commonsense reasoning. many recently developed benchmark datasets for commonsense reasoning formulate problems as multiple choice questions (mcqs): piqa @xcite , social iqa @xcite , com-monsenseqa @xcite , among others. the advantages of mcq evaluation are clear: with a single correct choice per question, system scores are easy to compute and understand. however, by their nature, commonsense reasoning questions typically do not have a single objectively correct answer; rather they admit many possible answers with varying degrees of plausibility as shown in figure 1 . under these conditions, what does it mean for a commonsense mcq answer choice to be the "correct" answer? we posit that the "correct" mcq answer in this setting should be the one that human annotators agree is most plausible among options. in principle, the plausibility of an individual mcq answer choice should depend only on the mcq context (if applicable), question, and the answer choice itself, but need not depend on the other answer choices. foot_1 under this assumption, then, a valid procedure to determine the correct mcq answer would be to rate the plausibility of each choice individually and select the highest-scoring option. in this paper, we analyze two important commonsense mcq benchmarks, social iqa (siqa; @xcite and commonsenseqa (csqa; tal- ## human data collection we select csqa @xcite and siqa @xcite for our study as they are popular mcq benchmarks for general commonsense and social commonsense reasoning, respectively. social iqa: mcq items consist of a short context describing a social situation, a question about a person in the situation, and three answer choices (see fig. 1 .) we randomly sample 125 questions from the validation split. these mcq items statistic siqa csqa original gold answer 3.86(0.73) 4.23(0.71) maximum rating 3.98(0.67) 4.33(0.63) second-best rating 2.88(0.74) 3.23(0.99) minimum rating 2.12(0.67) 1.43(0.47) maximum -second-best 1.10(0.77) 1.10(0.83) maximum -minimum 1.86(0.83) 2.90(0.67) commonsenseqa: mcq items consist of a question generated by humans using concept-net @xcite relations and five possible answer choices. we sample another 125 validation questions, which have gold labels based on approval by a second annotator after construction. for each of the 250 sampled mcq items from these two datasets, we collect two types of human judgments: individual plausibility ratings ( § 2.1) and full question annotations ( § 2.2). annotators are recruited through prolific and paid $15/hour; see appendix a.4 for details, including annotation interfaces (figure 5 and figure 6 ). annotation counts for the two tasks are presented in ## plausibly problematic mcqs with these collected judgments, we consider three ways to define a "correct" answer choice for each mcq item: (1) the original gold answer choices from siqa or csqa (y dataset ), (2) the majorityvote answer choice from full question annotation (y f ull ), and (3) the answer choice with the maximum mean plausibility rating (y plausibility ). we hypothesize that y plausibility should be predictive of y dataset and y f ull across mcqs, and that when they diverge it may be indicative of one or more problems with the underlying mcq. to corroborate this idea, first we show in figure 2a and figure 2b that a small difference in plausibility scores between the highest-and secondhighest scoring answers in the individual plausibility setting is correlated with lower agreement on the full question annotations, for both datasets. foot_3 this is consistent with the idea that disagreements on full mcq annotations may arise when there is not a clear most-plausible answer. next we compare y plausibility to y dataset . for both siqa and csqa, y plausibility diverges from y dataset in 22.4% of mcqs. we define these mcqs as "plausibly problematic" questions given that the answer choice selected as y plausibility did not match y dataset . ## implications for llm evaluation we prompt llms with the same task posed to humans in § 2.1 and § 2.2. we study multiple stateof-the-art llms: gpt-4 (gpt-4-0125-preview) @xcite with the openai api, llama-2 (7b, 13b and 70b) @xcite , mistral (7b and 7x8b) @xcite and yi (6b, 9b and 34b) @xcite . we prompt each llm with the same 10 in-context examples for the plausibility and full settings. we compare human and llm performance on the set of "plausibly problematic" and nonproblematic questions (identified using the plausibility ratings ( § 2.1)) and present the accuracy (against y dataset ) in plausibility in commonsense: ranking, comparing, and scoring the plausibility of events and outcomes expressed in language is a longstanding concept in commonsense reasoning research @xcite @xcite . because commonsense knowledge is often subjective (whiting and watts, 2024) or graded @xcite , and varies with cultural context @xcite @xcite , this can pose challenges for evaluation. most relevant to this work, @xcite use likert-scale human plausibility judgments of answer choices to construct cultural commonsense mcq test items. other approaches to evaluation include verbalized rationales @xcite . specifically, prior works have studied defeasible @xcite and abductive reasoning @xcite in natural language, where models rationalize when scenarios may be more plausible or valid. ## conclusion in this work, we show that plausibility judgments are a useful tool for identifying potentially problematic commonsense mcq items. with individual plausibility ratings, we are able to identify questions where the gold answer does not match the answer with the highest plausibility. through manual analysis we identify several types of issues that are more prevalent among the identified subset. we show that llms and humans perform poorly on these questions, with a high degree of variance, suggesting they add noise to benchmark evaluations. future work may investigate methods of incorporating plausibility judgments into the creation stage of benchmark development, as well as the application of these ideas to evaluating other types of benchmarks involving graded judgments beyond commonsense reasoning.