A Semantic-based Method for Unsupervised Commonsense Question Answering (2021)

Abstract: Unsupervised commonsense question answering is appealing since it does not rely on any labeled task data. Among existing work, a popular solution is to use pre-trained language models to score candidate choices directly conditioned on the question or context. However, such scores from language models can be easily affected by irrelevant factors, such as word frequencies, sentence structures, etc. These distracting factors may not only mislead the model to choose a wrong answer but also make it oversensitive to lexical perturbations in candidate answers. In this paper, we present a novel SEmantic-based Question Answering method (SEQA) for unsupervised commonsense question answering. Instead of directly scoring each answer choice, our method first generates a set of plausible answers with generative models (e.g., GPT-2), and then uses these plausible answers to select the correct choice by considering the semantic similarity between each plausible answer and each choice. We devise a simple, yet sound formalism for this idea and verify its effectiveness and robustness with extensive experiments. We evaluate the proposed method on four benchmark datasets, and our method achieves the best results in unsupervised settings. Moreover, when attacked by TextFooler with synonym replacement, SEQA demonstrates much less performance drops than baselines, thereby indicating stronger robustness.

## introduction pre-trained language models have been widely used for commonsense question answering. finetuning pre-trained models on task-specific data produces many state-of-the-art results @xcite * equal contribution † corresponding author: minlie huang. figure 1 : two examples of commonsense question answering, where the baseline (pro-a) is oversensitive to lexical perturbations (sr for synonym replacement and st for sentence structure transformation). the scores from pro-a and our method for each answer choice are shown in the right columns. the underlined score indicates the answer choice selected by a method. @xcite . however, this requires amounts of labeled task data. therefore, it is vital to study unsupervised commonsense question answering without relying on any labeled downstream task data. in this paper, we investigate multiple-choice commonsense question answering tasks in an unsupervised setting: given a question and a set of answer choices, a model is required to predict the most reasonable answer choice for the question, but without access to any labeled task data. many existing unsupervised methods tackle these tasks by scoring each answer choice using a language model, e.g., estimating the generative probability of the answer choice conditioned on the question @xcite @xcite . since existing methods can be easily distracted by irrelevant factors such as lexical perturbations, we argue that a commonsense question answering method should focus on the answers' semantics and assign similar scores to synonymous choices. to this end, we introduce a novel semantic-based question answering model, seqa, which aims to robustly select correct answers in multi-choice commonsense question answering in an unsupervised setting. instead of directly scoring an answer choice, we calculate the probability of observing the choice's semantics. a choice's semantic score can be obtained by summing the generative probabilities of sentences that have the same semantic meanings with the choice, where the sentences are called the choice's supporters. however, it is hard to obtain the supporters which have exactly the same semantic meanings with the choice, so we reformulate the semantic score into a soft version as explained in section 3.2. each supporter is weighed by the semantic similarity to the answer choice, which can be computed with some off-the-shelf models, such as sen-tencebert @xcite . since the supporters and their weights depend on the semantics rather than the surface form of the answer choice, by this means, the effects of the distracting factors can be largely suppressed. moreover, synonymous choices are likely to share the same set of supporters, so their scores are expected to be stably close. our contributions in this paper are summarized as follows: ## related work previous work has explored pre-trained language models (lms) for unsupervised commonsense question answering. in general, these approaches treat lms as question answering modules. some recent work claims that external knowledge can benefit commonsense reasoning. besides static knowledge bases (kbs), such as concept-net @xcite and atomic @xcite , there are also numerous studies treating lms as dynamic kbs. @xcite shows that lms can be used for kb completion. and davison et al. (2019) monsenseqa @xcite that generates explanations for questions, which are then used as additional inputs. the shortcoming of this approach is that it requires collecting human explanations for each new dataset to fine-tune lms. some following researches explore unsupervised explanation/knowledge generator. cga (bosselut and choi, 2019) employs comet @xcite to generate intermediate inferences which are then used to score the choice. however, comet is limited by a small set of question types so that cga is difficult to generalize to different domains. self-talk @xcite breaks the limit by extracting knowledge from gpt-2 (radford et al., 2019), which has no restriction on the query types. thus, self-talk can be applied to a wide range of domains. despite the introduction of auxiliary information, these methods are essentially dependent on language model scores, so they are still sensitive to lexical perturbations. besides directly using pre-trained lms, some recent efforts have been dedicated to automatically constructing task-specific data to train commonsense reasoners in zero-shot settings. @xcite and @xcite provide some rules to construct labeled training data from large corpus for pronoun disambiguation. banerjee and baral (2020), @xcite and @xcite collect training data based on knowledge bases, such as atomic @xcite . though effective, they are limited by the specific task settings or highly dependent on the task-related knowledge bases, which makes them difficult to transfer to other commonsense reasoning tasks. ## method in this paper, we focus on unsupervised multiplechoice commonsense question answering, which is formalized as follows: given a question and a set of choices, models should select the correct choice: ## conclusion we present a semantic-based question answering method, seqa, which can answer commonsense questions more accurately and robustly in an unsupervised setting. instead of directly scoring each answer choice, our method focuses on the probability of observing a choice's semantics. in the view of voting, seqa first generates some plausible answers (voters) and then utilizes them to vote for the correct choice by considering the semantic similarity between each choice and each voter. experiment results show that seqa achieves the best performance on four datasets, and it is remarkably more robust than all the baselines when being attacked by textfooler. tamborrino et al. (2020) provides templates for copa (