A Method for Building a Commonsense Inference Dataset based on Basic Events (2020)

Abstract: We present a scalable, low-bias, and low-cost method for building a commonsense inference dataset that combines automatic extraction from a corpus and crowdsourcing. Each problem is a multiple-choice question that asks contingency between basic events. We applied the proposed method to a Japanese corpus and acquired 104k problems. While humans can solve the resulting problems with high accuracy (88.9%), the accuracy of a high-performance transfer learning model is reasonably low (76.0%). We also confirmed through dataset analysis that the resulting dataset contains low bias. We released the datatset to facilitate language understanding research.

## introduction along with the progress of deep learning, there have been many studies that consider task settings and build their datasets for training/evaluating language understanding ability by computers @xcite . language understanding by computers requires two types of knowledge: knowledge of language (meaning of words, syntax, and so forth) and knowledge of our world and society beyond language. the former problem of acquiring linguistic knowledge has been solved to a large extent by general-purpose language models, such as bert @xcite , which are pre-trained using a large corpus. it is now possible to represent the meaning of a word as a vector according to its context. fine-tuning based on these vectors has made natural language inference, paraphrase recognition, and question answering without requiring deep inference as accurate as humans. on the other hand, there are still many problems with acquiring knowledge beyond language. actu-ally it is open-ended, and we had better start with fundamental knowledge, i.e., commonsense. still, it is not easy to focus on commonsense, guaranteeing some generality as commonsense. there have been some approaches to guarantee such generality. swag @xcite , for example, focuses on knowledge about daily events that can be visually perceived. this method greatly limits the range of commonsense that can be acquired. commonsenseqa @xcite is based on the basic vocabulary that is covered by conceptnet @xcite , which is one of the largest commonsense knowledge bases. this prevents the scalability, generating only 12k problems from the whole data of conceptnet. another important point is that biases in building datasets must be reduced as much as possible. in the above two approaches, question or distractor sentences were created automatically or by crowdsourcing. this causes generation biases of language models or produces certain patterns (annotation artifacts) by crowdsourced writing @xcite . we use a text corpus to solve these problems. we propose a method to build a commonsense inference dataset by extracting contingent pairs of basic event expressions (hereafter, contingent basic event pairs) from a corpus and verifying them by crowdsourcing. basic event expressions (hereafter, basic events) are defined as expressions composed of high-frequency predicate-argument structures that are extracted from a corpus and aggregated by clustering according to their usages. contingent basic event pairs are extracted by identifying contingency relations between basic events using discourse parsing. for instance, the following contingent basic event pairs are acquired. i am hungry, so a. i drink coffee. b. i have a meal. c. i sweat. d. i get sleepy. based on these contingent pairs, we can generate a commonsense inference problem by adopting latter events of other pairs as distractors, as shown in since our method is based on automatic extraction from a corpus, it is scalable and the domain is not limited. in addition, there is no bias caused by crowdsourcing because we ask crowdworkers to just verify a sentence. although the key idea of our proposed method is language-independent, we build a japanese commonsense inference dataset in this study by exploiting existing resources. the contributions of this paper are summarized as follows. ## related work language resources for commonsense inference that have been built so far can be classified into knowledge bases and qa datasets. commonsense knowledge bases have been constructed by experts, crowdsourcing, and games with a purpose. cyc @xcite and open mind common sense projects @xcite collected various relations between entities and events. they include causal relations between events, but the number of these relations is not high. atomic @xcite is a knowledge base that is comprised of 877k if-then pairs of basic events. they collected these pairs using crowdsourcing based on frequent basic events extracted from several corpora. these fully manual or crowdsourcing approaches are costly and have a problem of scalability. also, methods for incorporating such knowledge bases into an nlp model have been studied but have not been established yet. many qa datasets for commonsense inference have been built. they include copa (choice of plausible alternatives) @xcite , swag @xcite , hellaswag @xcite , and commonsenseqa @xcite . these datasets can be solved to some extent by machine comprehension models @xcite that have been rapidly improved. there have been also some approaches that transfer knowledge in such a dataset to downstream tasks using multi-task learning @xcite . we briefly introduce these datasets below. copa consists of 1,000 two-choice questions that ask a causal relation between two sentences. each question provides a premise sentence and requires to choose its cause or ending sentence from two alternatives. this dataset was manually created for the purpose of evaluation and is too small to learn commonsense by computers. swag is a commonsense inference dataset consisting of 113k multiple-choice questions that ask the most appropriate verb phrase following a given context. to guarantee generality as commonsense, questions were created from video captions, and thus the domain of the dataset is limited to the physical world. for each question, two consecutive sentences were extracted from a video caption, the first sentence and the subject of the second sentence compose a context, and the rest was regarded as a correct choice. distractors were generated from a language model. to obtain high-quality distractors, swag removed those that are easily discriminated by an answer model. swag was solved by bert with a similar accuracy to humans. this was attributed to biases that were embedded in distractor sentences by an lstm-based language model and detected by bert @xcite . they newly built hellaswag using a better language model to make biases undetectable by bert. however, the accuracy for solving hellaswag is also approaching to human performance @xcite . the bias problem has not been solved yet. commonsenseqa is a commonsense inference dataset consisting of 12k multiple-choice questions based on the commonsense knowledge base, con-ceptnet. a question is created by crowdsourcing based on a subgraph consisting of a source concept and three target concepts connected with the same relationship. a crowdworker creates a question sentence which includes the source concept and whose answer is only one of the target concepts. this method uses an existing resource and lacks scalability. in addition, because the load of creating question sentences is large for crowdworkers, they tend to use the same words and styles repeatedly, leading bias in question sentences. ## a method for generating commonsense inference problems a commonsense inference problem consists of a context (question) and four choices. the question asks to choose the most appropriate choice following the context, as shown in we describe the details of each step in the following subsections. ## building a japanese commonsense inference dataset we built a japanese commonsense inference dataset using the method described in section 3. ## experiments we conducted experiments to investigate the performance of a transfer learning model on the constructed commonsense inference dataset. ## conclusion in this paper, we proposed a scalable, low-bias, and low-cost method for building a commonsense inference dataset that combines automatic extraction from a corpus and crowdsourcing. each problem is a multiple-choice question that asks contingency between basic events. we applied the proposed method to a japanese web corpus and acquired 103,907 problems. while the human accuracy was high (88.9%), the bert large accuracy was reasonably low (76.0%). we also confirmed that the dataset contained low bias, and thus it can be used as a good benchmark for language understanding research. in the future, we will make a model learn commonsense with the obtained dataset and consider applying it to semantic tasks, such as anaphora resolution and discourse parsing. for commonsense acquisition from text, there is a problem that every commonsense is not written in text because of reporting bias @xcite . to acquire a wider range of commonsense, it is possible to combine our method with other methods based on physical world resources, such as video captions used in swag.