Zero-shot Event Causality Identification with Question Answering (2022)

Abstract: Extraction of event causality and especially implicit causality from text data is a challenging task. Causality is often treated as a specific relation type and can be considered as a part of relation extraction or relation classification task. Many causality identification-related tasks are designed to select the most plausible alternative of a set of possible causes and consider multiple-choice classification settings. Since there are powerful Question Answering (QA) systems pretrained on large text corpora, we investigated a zero-shot QA-based approach for event causality extraction using a Wikipedia-based dataset containing event descriptions (articles) and annotated causes. We aimed to evaluate to what extent reading comprehension ability of the QA-pipeline can be used for event-related causality extraction from plain text without any additional training. Some evaluation challenges and limitations of the data were discussed. We compared the performance of a two-step pipeline consisting of passage retrieval and extractive QA with QA-only pipeline on event-associated articles and mixed ones. Our systems achieved average cosine semantic similarity scores of 44 – 45% in different settings.

## introduction the aim of the work was to exploit the reading comprehension of pre-trained question answering (qa) models to address zero-shot event causality extraction from text. since implicit causality can be expressed in various, potentially infinite number of ways, and causality expressions can be distributed throughout sentences, identification of event causality remains a challenging task. many related data resources are designed for binary statement classification, multiple-choice qa, or relation classification. for our experiments we used a semantic similarity search-based dataset obtained from annotated wikipedia articles. the dataset was designed for event-related causality extraction from plain text. however, the data had some limitations discussed in the related section. we compared a two-step extraction pipeline consisting of relevant text passage retrieval based on semantic similarity search and cause candidate retrieval based on qa. the experiments were performed in two different settings: related documents and mixed documents subsets. the paper is structured as follows: section 2 overviews related work on causality identification, including some question-driven approaches. section 3 describes our data, experiments and evaluation metrics, and section 4 presents the results. ## results and discussion the evaluation results are summarized in the number of exact matches is very low (< 1%) in all cases. there is no significant difference between the two approaches, judging by the metrics. however, qa-only is more time-intensive because it processes all text chunks. the qa-only approach provided two and one more exact matches than passage retrieval + qa in related document-and mixed document-settings, respectively, as well as slightly higher f1-scores. in the mixed setting qa-only was able to find candidates for all annotations while the two-step approach missed candidates for three causes, i.e., some salient text passages were ignored during passage retrieval. this issue can be addressed by increasing the number of passages and/ or improving the quality of passage ranking techniques. however, we still think that the passage extraction step can have advantages when dealing with large text collections. further experiments are needed to prove this. the results could be improved by additional domain-specific model training and increasing the number of retrieved passages and answer candidates. generative summarization could be a better choice than using only extractive methods. the first example demonstrates a large gap between the low probability of being an answer to the asked question and the high score of matching with the ground truth annotation. in a real-world proceedings of clib 2022 application, relying on the qa score, this answer would be low-ranked. the second example can be considered satisfactory by human judgement. although the best answer conveys the main idea of the true annotation, its answer matching score is relatively low, as well as its qa score. the third example illustrates a cause candidate scored highly by the qa model but having a relatively low answer matching score. these examples demonstrate the need to define a sufficient level of similarity, because even similarity scores under 0.5 may still indicate adequate matches. ## conclusions and future work in this work, we conducted experiments to evaluate the zero-shot event causality identification with semantic search-based passage retrieval and qa on a dataset obtained from wikipedia. we compared the two-step and the qaonly approaches on related and mixed documents and demonstrated their similar performance in the experimental settings. while the two-step approach could not find any candidates for a few ground truth annotations in the mixed document setting, qa-only was able to find candidates in all cases. qa-only also performed slightly better on related documents, however, it required more computational time. further experiments are necessary to identify whether the passage retrieval step bring other advantages when processing large document collections. our systems achieved average cosine semantic similarity scores of 44 -45% in different settings. we think that the reading comprehension of qa models can be used to address the challenge of event causality extraction. in the future work, both passage and answer retrieval can be improved by using models with domain-specific knowledge, as well as increasing the number of retrieved passages and candidate. using other or multiple question templates could help to retrieve more various cause candidates.