CASE : Commonsense-Augmented Score with an Expanded Answer Space (2023)

Abstract: LLMs have demonstrated impressive zero-shot performance on NLP tasks thanks to the knowledge they acquired in their training. In multiple-choice QA tasks, the LM probabilities are used as an imperfect measure of the plausibility of each answer choice. One of the major limitations of the basic score is that it treats all words as equally important. We propose CASE, a Commonsense-Augmented Score with an Expanded Answer Space. CASE addresses this limitation by assigning importance weights for individual words based on their semantic relations to other words in the input. The dynamic weighting approach outperforms basic LM scores, not only because it reduces noise from unimportant words, but also because it informs the model of implicit commonsense knowledge that may be useful for answering the question. We then also follow prior work in expanding the answer space by generating lexically-divergent answers that are conceptually-similar to the choices. When combined with answer space expansion, our method outperforms strong baselines on 5 commonsense benchmarks. We further show these two approaches are complementary and may be especially beneficial when using smaller LMs.

## introduction large language models (llms) have demonstrated strong few-shot and zero-shot performance across various nlp tasks, with the larger models often matching earlier fine-tuned approaches that relied on task-specific labeled data @xcite @xcite . we focus on the zero-shot setup, which assumes that the knowledge needed to perform a specific task is already present in the llm @xcite @xcite . zero-shot learning has been employed for tasks such as translating between unseen language pairs @xcite , summarization @xcite , commonsense reasoning @xcite ; klein she wanted to sue her former employer . she decided to sue her employer . she decided to run for office . and nabi, 2021; @xcite , and more. in multiple-choice question answering (mcqa) tasks, zero-shot methods typically rely on the language model (lm) probabilities as a proxy for plausibility, predicting the answer choice with the highest probability conditioned on the question. lm score is a naïve proxy for plausibility, since it confounds factors such as length, unigram frequency, and more @xcite . indeed, in figure 1 , a gpt-2 based lm score incorrectly predicts that the woman hired a lawyer because she decided to run for office, rather than because she decided to sue her employer. in this paper, we propose to address one of the major limitations of the lm score. by summing or averaging the token-level probabilities, the lm score treats all tokens as equally important. a person reading this question would likely pay attention to option a because the word "sue" is highly relevant in the context of a lawyer. this signal might be weaker in a basic lm score where the word "sue" is conditioned on each other token in the question and previous tokens in the answer. furthermore, the lm might miss non-trivial connections between related words. to address this challenge, we propose case: a commonsense-augmented score with an expanded answer space. case is a post-hoc dynamic weight scoring algorithm that prioritizes important words in the sentence. the importance of each individual word is determined based on its relationship with other words in conceptnet @xcite . for example, conceptnet provides the information that "sue requires having a lawyer". we use the word-level importance scores to re-weigh the lm probability scores. indeed, in the second line of option a in figure 1 , the importance of the word "sue" increases the score of the entire sentence, leading to correctly predicting a as the correct answer. we further adopt the strategy suggested by @xcite to expand the answer space by using a lm to generate additional answers and then mapping semantically-similar generated answers into the original space. this mitigates the lm score's sensitivity to infrequent words. figure 1 demonstrates that a generated option c, "she wanted to sue her former employer", which is conceptually similar to a, further yields a higher probability score with our method. we tested case on 5 popular commonsense mcqa datasets. case outperformed the broad range of strong baselines that we compared with, confirming that it is an effective method for zeroshot mcqa. we further study the impact of different model sizes, answer candidates of varying qualities, and different weight assignment strategies on the performance. 1 1 our code is available at github. ## method we propose case, a commonsense-augmented scoring method with an expanded answer space. case can be used for zero-shot mcqa tasks. it is based on lm score (section 3.1). however, rather than treating all words in the context and answers as equally important, we propose a weighted score where the conditional probability is weighed by the importance of a word. the weights are determined using a commonsense kb in order to provide information that humans might implicitly be reasoning about when answering such questions (section 3.2). following @xcite , we expand the set of answer candidates by generating free-text answers, to increase the scorer's robustness to lexical variability (section 3.3). an overview of the method is shown in figure 2 . ## qualitative analysis we focus on case and look at the individual token scores and corresponding conceptnet paths to better understand the model decision-making process. figure 4 shows an example from sct where case predicted the correct answer. the word "upset" in the correct answer choice was assigned a high weight by case thanks to conceptnet paths such as upset related to ←---→ depression causes ←--stress related to ←---→ work. conversely, in figure 5 , case predicted the incorrect answer choice for another sct example. the model focused on the word "left" due to its semantic relation to the word "drove", failing to understand that priya drove to and not away from the restaurant. ## conclusion we presented case, a novel lm-based plausibility score for zero-shot mcqa tasks. case uses a commonsense kb to assign importance weights to words in the input. the weighting strategy outperforms basic lm scoring methods. when combined with generating additional answer candidates, case outperforms the baselines on 5 popular mcqa benchmarks. we further showed that the two approaches are complementary and are especially beneficial when using smaller lms. in the future, we plan to explore a more selective approach jim is a new driver and has never been pulled over before. yesterday he was pulled over for speeding. the officer explained to him why he was being given a ticket. jim will have to work overtime to pay for the ticket. jim is very upset. priya decided to try a new restaurant. she drove to a new cafe that opened. priya sat at a booth. she ordered a mimosa and a breakfast burrito. priya decided not to eat and left. for knowledge retrieval from the kb, and adapt case for additional nlp tasks.