Improving Commonsense Contingent Reasoning by Pseudo-data and Its Application to the Related Tasks (2022)

Abstract: Contingent reasoning is one of the essential abilities in natural language understanding, and many language resources annotated with contingent relations have been constructed. However, despite the recent advances in deep learning, the task of contingent reasoning is still difficult for computers. In this study, we focus on the reasoning of contingent relation between basic events. Based on the existing data construction method, we automatically generate large-scale pseudo-problems and incorporate the generated data into training. We also investigate the generality of contingent knowledge through quantitative evaluation by performing transfer learning on the related tasks: discourse relation analysis, the Japanese Winograd Schema Challenge, and the JCommonsenseQA. The experimental results show the effectiveness of utilizing pseudo-problems for both the commonsense contingent reasoning task and the related tasks, which suggests the importance of contingent reasoning.

## introduction contingency is the relation between two events, one being an action or state and the other being likely to happen after it. we humans reason contingent relation between events on a daily basis. for instance, when we read text, we unconsciously infer what happens next to deepen our understanding. in conversations, we guess the next topic from the utterance of the opponent to make a contextual and natural response. thus, the ability to reason contingent relation between events is essential when it comes to natural language understanding (nlu). recently, many studies have built language resources for contingent reasoning @xcite @xcite @xcite . these resources focus on basic events and evaluate some kind of commonsense reasoning ability. although the fundamental linguistic capabilities of comput-i'm hungry, so a. i'm gonna be absent from school. b. i refrain from strenuous exercise. c. i have a meal at a family restaurant. d. i leave home. ers, such as question answering, have greatly improved with progress in deep learning, several studies have empirically demonstrated they still have difficulty in commonsense reasoning @xcite @xcite . in this study, we aim at two objectives: to improve commonsense contingent reasoning and to investigate the effects of learning contingent knowledge on the related tasks to validate the importance of contingent reasoning. to these ends, we use the kyoto university commonsense inference dataset (kuci) foot_0 . kuci is a japanese qa dataset with 104k multiple-choice questions that ask contingent relation between basic events directly @xcite . an example is shown in figure 1 . this dataset is also characterized by its semi-automatic data construction method: automatic extraction of contingent pairs of basic event expressions from a web corpus, verification through crowdsourcing, and automatic generation of commonsense inference problems. it is shown there is a performance gap between humans and computers on this task @xcite . furthermore, through qualitative evaluation, it has been confirmed computers sometimes answer contingent relation between quite basic events incorrectly. one straightforward approach to alleviating the above problem is to extend the train-ing data and increase the coverage. however, it is not practical from a cost perspective to increase the number of training examples manyfold using crowdsourcing. we attempt to improve the performance by omitting crowdsourcing, a bottleneck in data augmentation, and utilizing pseudo-problems generated automatically from unverified contingent pairs of basic event expressions. as a web corpus is scalable, and all of the procedures except crowdsourcing are automatic, we can generate pseudo-problems at scale. it is expected pseudo-problems complement the lack of coverage though some problems are noisy and might be unanswerable. the second objective of this study is to investigate the effects of learning contingent knowledge on the related tasks. on the premise that contingent reasoning is essential to nlu, we can expect contingent knowledge probably helps improve the performance on other nlu tasks. while the transferability of major english datasets has been studied @xcite @xcite , there is room to explore this dataset in terms of the task and language. we investigate the generality of contingent knowledge through quantitative evaluation of transfer learning on the related tasks. in summary, we improve commonsense contingent reasoning by straightforward data augmentation. we generated 862k pseudo-problems, which is about ten times as large as the training examples in kuci (83k), and incorporated them into training. owing to pseudo-problems, a high-performance pre-trained model has achieved near human-level performance on the commonsense contingent reasoning task. we also investigate the transferability of contingent knowledge to the related tasks. our experiments demonstrate intermediate-task training on kuci with pseudo-problems positively affects discourse relation analysis, the japanese winograd schema challenge, and the jcommonsenseqa, which suggests the importance of contingent reasoning foot_1 . ## approach first, we describe our data augmentation approach to improving commonsense contingent reasoning. our approach is to automatically generate largescale pseudo-problems based on the construction contingent basic event pairs crowdsourcing contingent event pairs commonsense inference problems i am hungry, so i have a meal. if i have a meal, i get sleepy. since i am sleepy, i drink co ee. if i exercise hard, i sweat ⋮ parsed web text unveri ed contingent basic event pairs basic condition case frames be hungry have a meal it rains ⋮ core events ① ② ③ i am hungry, so a. i drink co ee. b. i have a meal. c. i sweat. d. i get sleepy. ④ reliable condition -our approach -generating pseudo-problems ③' a⇒b a⇏b figure 2: overview of the method of generating commonsense inference problems in kuci (gray) and pseudo-problems (red). the further details are described in omura et al. (2020). ## experiments we conducted experiments to investigate the effects of incorporating pseudo-problems into training on the commonsense contingent reasoning task and the related tasks. ## related work owing to large-scale pre-training, the pre-trained models have achieved unprecedented performance on a variety of nlu tasks, including commonsense reasoning @xcite . besides such improvement in general language understanding, there have been many approaches to improving the performance on commonsense reasoning tasks. one group of approaches is to utilize automatically created data, to which our approach belongs. for instance, @xcite performed additional pre-training on 16 million fill-in-theblank multiple-choice questions generated from wikipedia and conceptnet @xcite . they improved the performance on two benchmarks for entity-level commonsense reasoning, commonsenseqa @xcite and winograd schema challenge (wsc) @xcite , though their method requires the manually constructed resource (conceptnet). @xcite proposed a data augmentation method for the choice of plausible alternatives (copa) and its extension @xcite , which consists of roughly three steps: filtering web text by several conditions, extracting causal pairs of clauses with the clue of discourse connectives, and generating distractors using language models. they have not investigated the application to the related tasks, focusing on improving commonsense causal reasoning. as for the second objective of this study, there are several studies about the transferability of commonsense knowledge from existing language resources. for instance, it has been reported intermediate-task training on two benchmarks for commonsense reasoning, social iqa @xcite and winogrande @xcite , helps improve the performance on wsc and copa. @xcite showed the datasets that require complex commonsense reasoning, such as cosmosqa @xcite and hellaswag @xcite , are beneficial to several target tasks. @xcite ran multi-task learning on multiple resources for commonsense reasoning to examine their interactions. since they have used the datasets that require complex reasoning, they have not focused on a specific type of commonsense reasoning. we focus on commonsense contingent reasoning and investigate the transferability in the language other than english. ## conclusion in this study, we improved commonsense contingent reasoning by incorporating large-scale pseudoproblems into training. we automatically generated 862k pseudo-problems from a japanese web corpus of 3.3 billion sentences using the existing data construction method with modification. owing to pseudo-problems, a high-performance pre-trained model has achieved near human-level performance on the commonsense contingent reasoning task. we also investigated the effects of learning contingent knowledge on the related tasks: discourse relation analysis, the japanese winograd schema challenge, and the jcommonsenseqa. our experiments demonstrated intermediate-task training on kuci with pseudo-problems has a positive impact on the related tasks, which indicates the importance of contingent reasoning. we will further analyze what kind of problems current models still answer incorrectly. from the qualitative analysis, we consider building a language resource for evaluating deeper language understanding. as another research direction, it is also tempting to pursue the improvement in nlu by learning various discourse relations between entities or events in documents.