Unsupervised Commonsense Question Answering with Self-Talk (2020)

Abstract: Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pre-trained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on self-talk as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as “what is the definition of...” to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the self-talk induced knowledge even when leading to correct answers is not always seen as helpful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.

## introduction human level natural language understanding involves reading between the lines and relying on implicit background knowledge. consider the sentence: alice let bob stand in front of her at the concert. using physical and social commonsense -(i) bob and alice want to see the stage, and (ii) if bob is taller, they would block alice's view -one can infer that alice is taller than bob. such examples are ubiquitous across natural language understanding (nlu) tasks such as reading comprehension @xcite and recognizing textual entailment @xcite , and even more so in tasks dedicated to commonsense reasoning such as the winograd schema challenge @xcite . most current nlu models rely on pretrained language models (lms; e.g. @xcite @xcite . the standard practice is to fine-tune a pre-trained lm in a supervised manner on task-specific data. alternatively, lm score is used to rank answer choices in a zero-shot setup @xcite . in both setups, pre-trained lms yield improved performance upon prior methods, greatly due to the world knowledge that such lms capture, having been trained on massive texts @xcite . despite the performance boost, lms as knowledge providers suffer from various shortcomings: (i) insufficient coverage: due to reporting bias, many trivial facts might not be captured by lms because they are rarely written about @xcite . (ii) insufficient precision: the distributional training objective increases the probability of non-facts that are semantically similar to true facts, as in negation ("birds cannot fly"; @xcite . lms excel in predicting the semantic category of a missing word, but might predict the wrong instance in that category (e.g., depending on the phrasing, bert sometimes predicts red as the color of a dove). finally, (iii) limited reasoning capabilities: it is unclear that lms are capable of performing multiple reasoning steps involving implicit knowledge. to increase the coverage of high-precision world knowledge and facilitate multi-hop reasoning by making intermediate reasoning steps explicit, prior work incorporated kbs (e.g. conceptnet; @xcite and knowledge-informed models into lm-based models @xcite @xcite . in this paper, we study pre-trained lms as an alternative to external kbs in providing knowledge because brett found an internship while in college but ian was unable to, brett found a job less quickly after graduation. the purpose of the internship is to help people find jobs. s 11 min i (s i2 ) because brett found an internship while in college but ian was unable to, ian found a job less quickly after graduation. the purpose of the internship is to help people find jobs. ## tasks we focused on the multiple-choice question answering tasks detailed below. each instance consists of an optional context, an optional question, and several answer choices. copa: choice of plausible alternatives @xcite : asking about either a plausible cause or a plausible result, among two alternatives, of a certain event expressed in a simple sentence. commonsenseqa: commonsense question answering @xcite : general questions about concepts from conceptnet. to increase the challenge, the distractors are related to the target concept either by a relationship in conceptnet or as suggested by crowdsourcing workers. ## models a given instance consists of an optional context c, an optional question q, and answer choices: a k @xmath0. we first describe the baseline model, which makes taylor was doing her job so she put the money in the drawer. job, money job money work type of mo tiv ate d by go al job is a type of work. you would work because you want money. job to earn money. what will taylor do next? xwant as a result, taylor wants to keep the money in the drawer. the prediction based on the instance alone ( §3.1). ## results best knowledge source. among the knowledge informed models, comet achieves the best performance across tasks. this likely happens because comet can dynamically generate predictions for any context, while the other two knowledge sources are static and lack coverage. we also experimented with combining the clarifications from all the knowledge sources, which didn't prove beneficial except for mc-taco (where it added +7.9 points to the dev accuracy, bringing it to 66.7). we assume that some resources added noise, making the whole smaller than the sum of its parts. ## analysis while the performance on the end task serves as an extrinsic evaluation for the quality of the generated clarifications, we are also interested in evaluating it intrinsically. from preliminary experiments we know that there is a high ratio of noisy clarifications. we thus focus on and analyze two types of clarifications: useful ( §5.1) and harmful ( §5.2). foot_4 ## discussion and conclusion we presented an unsupervised framework for multiple choice commonsense tasks that generates and integrates background knowledge from pre-trained lms. on most tasks, it performs substantially better than the baseline and similarly to a model that had access to external knowledge resources. we have listed several shortcomings of using pre-trained lms as knowledge providers: (i) insufficient coverage, (ii) insufficient precision, and (iii) limited reasoning capabilities. despite their insufficient precision compared to a kb like con-ceptnet, we showed that clarifications generated by lms resulted in similar or superior empirical gains. among the clarifications used in practice by the answer scorer, about 60% of those that yielded a correct prediction and 12% of those that yielded an incorrect prediction were judged by humans as factually correct. by design, our model makes a single additional reasoning step explicit, aiming to facilitate reasoning about implicit inferences. a preliminary experiment in which we incorporated clarification pairs to facilitate two hops got mixed results. an interesting future direction is to generate each clarification in response to the previous ones, in a dialogue setup @xcite . another challenge is the "needle in a haystack" problem of the clarifications, and one way to address it is to develop a model that is capable of "introspection", specifically knowing what it doesn't know. a more structured knowledge generation might also make the combination of various knowledge sources more successful. filling in knowledge gaps and making implicit intermediate reasoning steps explicit is imperative going forward. we hope that our framework will facilitate future research in this area. our code and data will be made available upon publication. our code and data is available at github.com/vered1986/self talk. 0% 25% 50% 75% 100% c o m e t c o n c e p t n e t g o o g l e n g r a m s g p t d i s t i l g p t -2 g p t -2 s g p t -2 m g p t -2 l g p t -2 x l x l n e t -b a s e x l n e t -l other wrong sense restating the instance correct factually incorrect relevant nonsensical or ungrammatical irrelevant figure 9: types of errors caused by the harmful clarifications, for each knowledge source, across all tasks.