ESTER : A Machine Reading Comprehension Dataset for Reasoning about Event Semantic Relations (2021)

Abstract: Understanding how events are semantically related to each other is the essence of reading comprehension. Recent event-centric reading comprehension datasets focus mostly on event arguments or temporal relations. While these tasks partially evaluate machines’ ability of narrative understanding, human-like reading comprehension requires the capability to process event-based information beyond arguments and temporal reasoning. For example, to understand causality between events, we need to infer motivation or purpose; to establish event hierarchy, we need to understand the composition of events. To facilitate these tasks, we introduce **ESTER**, a comprehensive machine reading comprehension (MRC) dataset for Event Semantic Relation Reasoning. The dataset leverages natural language queries to reason about the five most common event semantic relations, provides more than 6K questions, and captures 10.1K event relation pairs. Experimental results show that the current SOTA systems achieve 22.1%, 63.3% and 83.5% for token-based exact-match (**EM**), **F1** and event-based **HIT@1** scores, which are all significantly below human performances (36.0%, 79.6%, 100% respectively), highlighting our dataset as a challenging benchmark.

## introduction narratives such as stories and news articles are composed of series of events @xcite . understanding how events are logically connected is essential for reading comprehension @xcite . for example, figure 1 illustrates several pairwise relations for events in the given passage: "the deal" can be considered as the same event of "paramount purchased dreamworks," forming a coreference relation; it is also a complex event that contains "assumed debt," "gives access" and "takes over projects" as its subevents. the event "sought after" is facilitated by a previous event "created features." by capturing these event semantic relations, people can often grasp the gist of a story. therefore, for machines to achieve human-level narrative understanding, we need to test and ensure models' capability to reason over these event relations. in this work, we study five types of event semantic relations: causal, sub-event, co-reference, conditional and counterfac-tual, and propose to use natural language questions to reason about event semantic relations. figure 2 shows example question-answer pairs for each relation type. although previous works studied some subset of these relations such as sub-event @xcite , causal and conditional @xcite @xcite , most of them adopted pairwise relation extraction (re) formulation by constructing (event, event, relation) triplets and predicting the relation for the pair of events. event relations of re formulation are rigidly defined as class labels based on expert knowledge, which could suffer from relatively low inter-annotator agreements @xcite and may not be the most natural way to exploit the semantic connections between relations and events in their context. we instead propose to reason about event semantic relations as a reading comprehension / question answering task. natural language queries ease the annotation efforts in the re formulation by supplementing expert-defined relations with textual prompts. when querying causal relations, we can ask "what causes / leads to event a?" or "why does a happen?"; when reasoning sub-event relation, we can ask "what are included in event b?" or "what does b entail?" etc. "lead to," "included" and "entail," as textual cues, can help models better understand which relation is being queried. our question-answering task also poses unique challenges for reasoning event semantic relations. first, the correct answers can be completely different with slight changes of queries. in figure 2 , if we modify the third question to be "what would happen if europe supported albania?" then "oust president sali" becomes an invalid answer. this challenge allows us to test whether models possess robust reasoning skills or simply conduct pattern matching. second, answers must be in the form of complete and meaningful text spans. for the counterfactual example in figure 2 , a random text span "president sali berisha" is not a meaningful answer while a shortened answer "oust" is not complete. to get correct answers, models need to detect both event triggers and their event arguments. finally, there could be multiple valid events in a passage that can answer a question, and a good system should be able to identify different valid answers simultaneously as in the sub-event qa of figure 2 . these challenges make our task more difficult than the classification tasks in re. a few noticeable event-centric mrc datasets have been proposed recently. torque @xcite and mctaco @xcite are two recent mrc datasets that study event temporal relations. however, knowing only the temporal aspect of events could not solve many important event semantic relations. for example, in figure 1 , to understand that "assumed debt," "gives access" and "takes over projects" are sub-events of "the deal," a model not only needs to know that all these four events have overlapped time intervals but also share the same associated participants for "the deal" to contain the other three. we summarize our contributions below. 1. we introduce ester, the first comprehensive mrc / qa dataset for the five proposed event semantic relations by adopting natural language questions and requiring complete event spans in the passage as answers. 2. by proposing a generative qa task that models all five relations jointly and comparing it with traditional extractive qa task, we provide insights on how these event semantic relations interplay for mrc. 3. our experimental results reveal sota models' deficiencies in our target tasks, which demonstrates that ester is a challenging dataset that can facilitate future research in mrc for event semantic relations. ## definitions composing event-centric questions and answers requires identifications of both events and their relations. in this section, we describe our definitions of events and five event semantic relations. ## related work we briefly survey related work in this section in order to provide broader background over the two key components of ester: 1) event semantic relations and 2) event-centric reading comprehension. ## data collection in this section, we show our data collection procedure and describe the details of our approach to control annotation quality, including qualification exams and steps to validate and train workers. ## data analysis our passage preparation (section 4.1) produces 4.3k passages in total with 1887 of them randomly selected and annotated. we collect 6018 questions from 70 workers using amazon mechanical turk and 1471 of them fully validated by co-authors as the evaluation set. we further split our evaluation data into dev and test sets based on passages. the remaining data are used as the training set. a summary of data statistics is shown in ## experimental setup we design experiments to provide benchmark performances and understand learning challenges to facilitate future research on ester. we formulate our qa task as a conditional answer generation problem. this choice is inspired by recent works such as unifiedqa @xcite ) that achieve impressive outcomes by integrating various qa tasks (extractive, abstractive and multiplechoice) as a single generative qa pre-training task. @xcite and @xcite also show that by reformulating original extractive tasks as generation tasks, it enables models to better exploit semantic relations between context and labels as well as the dependencies between different outputs. to better demonstrate the benefits of the proposed generative qa task, we compare it with a traditional extractive qa task. we introduce our experimental design and evaluation metrics subsequently. ## results and analysis in this section, we present and analyze results for the experiments described in section 6. ## conclusion we propose ester, an mrc datasets for comprehensive event semantic reasoning. we adopt meticulous data quality control to ensure annotation accuracy. ester enables a generative question answering task, which can be more challenging than the traditional event relation extraction work. the difficulty of the proposed data and task is also manifested by the significant gap between machine and human performances. we thus believe that es-ter would be a novel and challenging dataset that empowers future event-centric research. 1. both questions and answers must contain correct events. events in questions can have different textual form.