BUCA : A Binary Classification Approach to Unsupervised Commonsense Question Answering (2023)

Abstract: Unsupervised commonsense reasoning (UCR) is becoming increasingly popular as the construction of commonsense reasoning datasets is expensive, and they are inevitably limited in their scope. A popular approach to UCR is to fine-tune language models with external knowledge (e.g., knowledge graphs), but this usually requires a large number of training examples. In this paper, we propose to transform the downstream multiple choice question answering task into a simpler binary classification task by ranking all candidate answers according to their reasonableness. To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts. Extensive experimental results show the effectiveness of our approach on various multiple choice question answering benchmarks. Furthermore, compared with existing UCR approaches using KGs, ours is less data hungry.

## introduction commonsense reasoning has recently received significant attention in nlp research @xcite , with a vast amount of datasets now available @xcite @xcite @xcite . most existing methods for commonsense reasoning either fine-tune large language models (lms) on these datasets @xcite or use knowledge graphs (kgs) @xcite to train lms @xcite . however, it is not always possible to have relevant training data available, it is thus crucial to develop unsupervised approaches to commonsense reasoning that do not rely on labeled data. in this paper, we focus on the unsupervised multiple choice question answering (qa) task: given a question and a set of answer options, the model is expected to predict the most likely option. we john wanted to be a better dancer. john wanted to be social with their friends. ## related work language models are widely used in unsupervised commonsense inference tasks, e.g. as an additional knowledge source or as a scoring model. @xcite propose an explanation generation model for the commonsenseqa dataset. self-talk @xcite uses prompts to stimulate gpt and generate new knowledge. seqa @xcite generates several candidate answers using gpt2 and then ranks each them. another research direction in unsupervised commonsense reasoning is the use of e.g. commonsense kgs @xcite @xcite to train the model @xcite . in @xcite , given the inputs of context, question and answer, the model learns to generate one of the inputs given the other two. @xcite update the model with a margin ranking loss computed on positive and negative examples from kgs. mico @xcite uses the distance between the positive and negative question-answer pairs obtained from the kg to calculate the loss. however, all of the above approaches demand a large amount of training data, sometimes reaching million of training samples, while buca only needs tens of thousands, cf. ## methodology we focus on the following multiple choice question answering (qa) task: given a question q and a set of options a, the model should select the most likely single answer a @xmath0. we consider an unsupervised setting in which the model does not have access to the training or validation data. our buca approach first trains the model with a knowledge graph and then uses the trained model to test on multiple qa downstream tasks. formally, a knowledge graph (kg) @xcite g is a tuple (v, r, t ), where v is a set of entities, e is a set of relation types and t is a set of triples of the form (h, r, t) with h, @xmath1. our approach has three main components: knowledge graph transfer to training data, training loss design, and downstream task testing: converting triples into binary classification training data. inspired by previous work @xcite , each kg triple is converted into question-answer pairs by using pre-defined templates, so that the obtained pairs are then used as the input of the classification task. we use the templates provided in @xcite . for example, the atomic triple (personx thanks persony afterwards, isafter, personx asked persony for help on her homework) can be converted to "after per-sonx asked persony for help on her homework, per-sonx thanks persony afterwards". in the appendix we show the distribution of the converted sequence pairs. along with the correct qa pairs created from the kg triples, our framework is also trained on negative qa pairs, so it can better discriminate between reasonable and unreasonable qa pairs. more precisely, in the training dataset, each correct qa pair generated from a triple @xmath2, r, t) has a corresponding negative pair obtained from a variation of tp in which t is substituted by t ′ , which is randomly drawn from the existing tails in the kg. training loss. for our binary classification model, we add a classification head with two nodes to the pre-trained language model. after normalizing the values on these two nodes, we can obtain reasonable and unreasonable scores for the qa pairs. from the triple conversion step, we obtained n training examples, each consisting of a question q, correct answer a c , and incorrect answer a w . for each question-answer pair, we can then obtain the reasonable and unreasonable scores r + i and r - i after applying a softmax layer. in each loss calculation, we jointly consider the correct and incorrect answers. for binary classification, we use two kinds of losses: traditional binary loss (tbl). in order to pull the representational distance between reasonable question-answer pairs as close as possible and to push the representational distance between reasonable and unreasonable ones as far as possible, we use supervised contrastive learning @xcite along with the binary classification. this is done by considering as positive examples of a given example within a category, all those examples within the same category. inference. in the prediction phase for each candidate answer, we calculate its reasonableness score. we choose the answer with the highest reasonableness score as the predicted answer. ## experiments in this section, we first describe our experiments on five commonsense question answering datasets, followed by ablation studies and data analysis. ## conclusion we presented a framework converting kgs into positive/negative question-answer pairs to train a binary classification model, discriminating whether a sentence is reasonable. extensive experiments show the effectiveness of our approach, while using a reasonably small amount of data. for future work, we will explore how to better select negative cases.