Every Answer Matters: Evaluating Commonsense with Probabilistic Measures (2024)

Abstract: Large language models have demonstrated impressive performance on commonsense tasks; however, these tasks are often posed as multiple-choice questions, allowing models to exploit systematic biases. Commonsense is also inherently probabilistic with multiple correct answers. The purpose of “boiling water” could be making tea, cooking but also could be killing germs. Existing tasks do not capture the probabilistic nature of common sense. To this end, we present commonsense frame completion (CFC), a new generative task that evaluates common sense via multiple open-ended generations. We also propose a method of probabilistic evaluation that strongly correlates with human judgments. Humans drastically outperform strong language model baselines on our dataset, indicating this approach is both a challenging and useful evaluation of machine common sense.

## introduction most existing commonsense evaluations utilize multiple-choice question-answering (mcqa) tasks @xcite @xcite @xcite . this format offers a limited view of common sense. mcqa tasks simplify the problem with unrealistically small answer sets, and making the options challenging is difficult @xcite . more crucially, common sense is implicit -understanding assumptions that are unspoken precisely because they are common knowledge. mcqa, by evaluating common sense explicitly, fails to capture a model's ability to utilize this knowledge in unprompted, generative contexts. mcqa is also fundamentally at-odds with the fact figure 1 : typical evaluations only compare human and model performance for their top choices (top). we propose to evaluate multiple plausible answer choices by clustering similar answers, denoted by color, to form categorical distributions, and evaluating probabilistically (bottom). this more accurately captures the probabilistic nature of common sense, and allows us to provide a more nuanced analysis of model capabilities. in order to avoid the issues in mcqa, many recent benchmarks have proposed generative commonsense evaluations. @xcite . while generative evaluation avoids the difficulty of generating hard negatives, it does not reflect the fact that there are often multiple correct answers, especially for commonsense questions. for example, in the phrase "they boiled the water", we can infer using our common sense that the most likely reason for this action is cooking or making tea. however, people in areas with limited clean water access may view this as a way to remove germs and ensure it's safe to drink. this aspect is, unfortunately, frequently overlooked during the benchmark creation process. to ensure that the model can serve diverse populations, it is important to gather multiple responses. by focusing on collecting implicit information from larger popu-lation, this provides a more accurate evaluation of common sense required in real-world settings. in this setting, it is crucial to address the commonsense questions with multiple correct answers. while some previous works have proposed a clustering-and-ranking evaluation @xcite , we demonstrate that such an approach can lack nuance. instead, we embrace the probabilistic nature of common sense and evaluate the model's ability to capture a probability distribution over answer clusters sampled from the population. this ensures that the model not only captures common sense, but does so in a calibrated way. to this end, we propose commonsense frame completion (cfc), a task focused on inferring missing information in a context sentence. this emphasizes the implicit nature of common sense in a given context and is tightly connected to downstream applications, such as home assistants, where inferring such information about a user's query is key. in cfc, the questions are generated by identifying missing information from a given context sentence about daily scenarios. for each context-question pair, we collected a large number of diverse answers from human annotators. in order to evaluate multiple responses effectively, we additionally propose a new evaluation method. responses from different individuals often vary, and common sense is typically defined as the knowledge shared by nearly all people. to make every answer count, we consider the annotators' answers from a probabilistic viewpoint and evaluate models using probabilistic measures @xcite @xcite . specifically, given raw string answers, we cluster them and consider the categorical distribution over these clusters based on the number of answers contained in each. we propose automated clustering and alignment mechanisms which allow models to be evaluated directly by comparing the kl divergence between distributions. the task format and evaluation metric are shown in figure 1 . we evaluate the proposed metric with two datasets, cfc and protoqa @xcite , and show high correlations with human judgments for both datasets. finally, we report multiple llms' performance on cfc measured by the proposed probabilistic metrics. we identify a large performance gap between existing large models and humans, indicating the limitations of current llms. ## related work commonsense evaluation creating commonsense benchmarks to evaluate model performance is a long-standing research topic @xcite @xcite . however, most benchmarks are created using a multiple-choice selection paradigm, which is simpler to evaluate but misaligned with the realworld use-case of commonsense knowledge, and most egregiously ignores the existence of multiple correct answers. we are not the first ones to gather multiple human answers to facilitate robust evaluations, however. aydin et al. (2014) and boratko* et al. (2020) also collected multiple human responses for each question to get aggregated human ground-truth answer sets. our work differs from these due to our emphasis on commonsense as implicit and probabilistic. we don't treat each answer equally; rather, we aim to match the answer distribution given by human responses. for this purpose, we propose a novel probabilistic evaluation for open-ended generation tasks with multiple correct answers. a similar probabilistic evaluation was studied from a language model generation point of view @xcite . they proposed a kl-based evaluation to measure language model generations, while our focus is on the implicit answer distribution. among the previous work, our proposed dataset cfc is most similar to protoqa @xcite as they share a similar task format (one question, multiple answers). however, there are several key differences. cfc is designed to uncover implicit commonsense information in various contexts, making it a more generalized variant of pro-toqa. consequently, the question format in cfc is significantly more diverse than that of protoqa. in addition, the number of answers collected for cfc is rigorously justified through the neyman-pearson lemma, guaranteeing a sufficient and representative sample. commonsense as probabilistic knowledge in most knowledge evaluation benchmarks, commonsense knowledge is defined as absolute facts @xcite . we relax this absolute intersection between human knowledge using a probabilistic approach. the probabilistic notion of knowledge is well supported. @xcite stated human beliefs or credences, inherently probabilistic, should be regarded as legitimate forms of knowledge. the growing trend towards building probabilistic models in cognitive studies reinforces the idea of human cognition and memory functioning as probabilistic processors @xcite . this collective body of work supports the view that common sense serves as probabilistic assumption instead of definite judgements in human mind. ## cfc task description in this section we motivate and describe the task of "commonsense frame completion" (cfc). we aim to create a task which evaluates implicit common sense with multiple correct answers. given a direction such as "put the water on the burner to boil", it is common sense which allows us to understand that the water is likely in a kettle and not simply dumped on the burner. unlike factual question answering tasks, there is no single correct answer. in this example, the water could be placed in a "kettle", "pot", "cup", or "glass", although the former answers are more probable. it is necessary for any machine learning model which claims to capture common sense to correctly predict all the possible answers and have some sense of the distribution over the implicit information. to assess a model's ability in this regard, we view the context sentence as a structured semantic frame, identify a missing slot, and ask the model to provide a distribution of potential slot fillers. ## dataset creation and analysis we now describe the dataset creation process of cfc. we first need to collect reasonable context sentences which contain natural element of common sense. commongen (lin et al., 2020) is a commonsense dataset which contains many short sentences describing basic information about daily life, and so we choose this dataset as the source for potential context sentences. given a short sentence, we then identify implicit information. to this end, we perform semantic parsing on the sentence, and identify missing slots. we use amr @xcite for semantic parsing based on its ability to provide a rich representation of the sentence with a pre-defined fixed schema for the predicate roles. if a predicate is found, amr parsing will match it to a schema and fill in the values for any identified slots. any slots marked with amr-unkown indicate potential items of missing information, enabling us to obtain human annotations for the missing slot values. we uniformly sampled 63,788 sentences from the commongen dev dataset, and parsed them using the amr parser from cai and lam (2020), generating 228,170 pairs of context questions with missing slots. from this, we randomly sampled 101 (sentence, missing slot) pairs for crowd workers to annotate, such that we had a balanced distribution of missing slot types, as detailed in section 4.2. we present the context sentence and missing slot to crowdworkers, who were also provided with training examples and descriptions of the meaning of each slot type (see ## probabilistic evaluation in this section, we detail the method of evaluating multiple correct answers. as we relaxed commonsense to be probabilistic knowledge, a rigorous probabilistic evaluation is required; however the task is presented (both to humans and models) as a generative question answering task. therefore, we need a way to compare two large sets of answer strings. we will start by how human evaluators may compare the similarity of there sets of answers and then describe the various ways by which this process can be automated. ## cfc results given the high correlation of probeval with human gold kl scores, we employ probeval in evaluating cfc model performance. all the evaluator parameter are tuned on cfc dev data, then fix the parameters to report results on cfc test data. ## conclusion in this paper, we assert that commonsense is an implicit probability distribution over missing information, and propose a dataset that aims to evaluate commonsense in this setting via a generative question answering task; moreover, we embrace the probabilistic nature of commonsense knowledge in both the dataset creation and the metric design. we propose a probabilistic automatic evaluation probeval for evaluating answer distributions that is highly correlated to human judgment. using this metric, we observe that model performance on our new dataset is significantly worse than human performance, indicating that the task is sufficiently challenging. in the future, we aim to further extend the size of the dataset, both in number of instances as well as answer length.