Narrative Why-Question Answering: A Review of Challenges and Datasets (2022)

Abstract: Narrative Why-Question Answering is an important task to assess the causal reasoning ability of systems in narrative settings. Further progress in this domain needs clear identification of challenges related to understanding the causal structure of narration. In this paper, we give an overview of the challenges related to both narrative understanding and why-question answering, because Narrative Why-Question Answering combines the characteristics of these domains. We also identify narrative QA datasets containing why-questions and analyze their characteristics through the lens of these challenges.

## introduction narrative why-question answering is the task of answering why-questions in narrative settings. this task combines the challenging properties of both why-question answering and narrative understanding. as such, narrative why-question answering makes a suitable task for evaluating complex comprehension abilities of language models. why-question is one of the most challenging non-factoid question types @xcite because it requires discovering explicitly or implicitly stated causal relations from text. as such, why-questions can be used to test the causal reasoning abilities of qa systems. on the other hand, narratives can be considered a desirable testbed for machine reading comprehension (mrc) tasks because narratives play a central role in the life of human beings, they have implicit nature and complex structure, and fictional narratives are self-contained @xcite . although humans can easily identify causal relations in narratives and make inferences by using their background knowledge and by paying close attention to the timeline, cause, and motivation of the events/entities, current qa systems have difficulties extracting correct relations and making such complex inferences in narratives @xcite . in order to make further progress in the narrative why-question answering, one should be knowledgeable about challenges that exist in this domain and in its datasets. these challenges can stem from both the properties of the narrative understanding and the specifics of why-question answering. some previous works @xcite have mentioned commonsense-related challenges. however, we are not aware of any previous work that has attempted to give a comprehensive list of challenges related to this topic. in this paper, our goal is to give a wider overview of the potential challenges in narrative why-question answering that can help to inform the researchers working in this domain. furthermore, in terms of datasets, the tellmewhy dataset @xcite is the only dataset that solely focuses on narrative why-question answering. in this paper, we also address why-questions in multiple-choice, free-form, and extractive narrative qa datasets in order to more fully identify the scope of challenges in this domain. we believe that considering other narrative why-question answering datasets can also help further development in this domain. we start by reviewing the concepts and challenges of why-questions and narratives in sections 2 and 3. in section 4, we identify the datasets relevant to narrative why-question answering, and provide an overview of the commonly used evaluation measures. finally, in section 5, we analyze the dataset-and task-specific challenges according to the concepts mentioned in previous sections. ## why-questions a why-question is typically asked about a causal relation in the text. causal questions can be constructed in several ways and they are not limited to why-questions only. causal questions can be also asked with what (e.g., what is the cause of), which (e.g., which are the consequences of), and how (e.g., how dangerous is) @xcite . however, whyquestion is the only question type that solely represents causality and can be used to test causal reasoning @xcite @xcite . furthermore, answering why-questions requires more complex reasoning than answering other types of causal questions because it is more difficult for qa-systems to decide directly from a why-question which type of information needs to be searched for in the text @xcite . in the following subsections, we start by elaborating on the notion of causality, including how causal relations are expressed in the texts, and then review what ambiguities why-questions hold in relation to causality. ## narratives narratives are texts in which events are causally or thematically linked and develop within a temporal framework @xcite . narratives are generally agent-oriented and their main scope is centered on characters, their actions, and motivations @xcite . in narrative qa, stories, fairytales, books, and (movie) scripts are commonly utilized as narrative texts. characteristics of narrative texts, such as causality of events and motivations of agents, make narratives a suitable context for asking why-questions. additionally, fictional narratives can ensure the test of comprehension because they are self-contained, meaning that all elements needed to understand the narrative, such as events, characters, and settings, are present in the text and qa models need to comprehend the narrative in order to answer questions @xcite @xcite . implicitness is a key feature of narratives that makes it different from other types of texts. length is another characteristic dimension of narratives which is also very important for qa systems. in the following subsections, we will review these characteristics in more detail. ## narrative why-question answering narrative why-question answering task can be formulated as a special case of why-qa where the context is a complex structured text-a narrative. currently, there exists only one qa dataset (tellmewhy) that solely consists of why-questions in narrative setting. additionally, several other narrative qa datasets contain why-questions in various proportions. the subsets consisting of whyquestions can be extracted from these datasets and used for training or testing narrative why-question answering systems. in the following subsections, we first review these potential datasets suitable for narrative why-qa, and then give an overview of common evaluation measures used to assess the performance of narrative why-question answering systems. ## challenges in this section, we list challenges based on our analysis of the datasets and the characteristics of the narrative why-question answering. this section is divided into three parts. in the first part, we focus on the commonly addressed challenges in the narrative why-question answering. the existing datasets are created generally to test the abilities of models on these challenges. in the second part, we review some potential challenges that are not marked in existing datasets and that the current models thus cannot be tested on. lastly, we review challenges that stem from both the datasets and the characteristics of the narrative why-question answering that can create problems on benchmarking models on this task. ## conclusion in this paper, we reviewed challenges and datasets related to narrative why-question answering. the challenges that occur in this domain can stem from both the properties of narrative understanding and the specifics of why-question answering. the main challenges regarding narrative understanding are the exclusion of common knowledge, the necessity of understanding the local and global narrative structure, and high-level abstraction. the primary challenges of why-question answering are related to identifying causal relations, and ambiguities in questions and answers. in order to understand data-specific challenges and the implications of task-specific challenges in datasets, we reviewed seven datasets that can be suitable for this task. we listed challenges that models are tested on in these datasets. the implicitness of questions and the length of narratives are more tested challenges in the datasets. we outlined different why-question formats and questions about unmarked and ambiguous causal relations as other potential challenges that models can be tested on. finally, we argued that general questions, and answer and question ambiguities in why-questions can create challenges for benchmarking. we propose that removing general questions and resolving ambiguities in why-questions can lead to more accurate evaluation of systems. we hope that the review of this potential challenges and the analysis of the datasets listed in this paper will help to further the progress in narrative why-question answering.