On Narrative Question Answering Skills (2024)

Abstract: Narrative Question Answering is an important task for evaluating and improving reading comprehension abilities in both humans and machines. However, there is a lack of consensus on the skill taxonomy that would enable systematic and comprehensive assessment and learning of the various aspects of Narrative Question Answering. Existing task-level skill views oversimplify the multidimensional nature of tasks, while question-level taxonomies face issues in evaluation and methodology. To address these challenges, we introduce a more inclusive skill taxonomy that synthesizes and redefines narrative understanding skills from previous taxonomies and includes a generation skill dimension from the answering perspective.

## introduction narrative question answering entails responding to questions based on a narrative context. understanding narratives requires comprehension of the foundational narrative elements @xcite that are not only explicitly stated but also implied in the text, which necessitates "reading between the lines" @xcite . additionally, answering questions in narrative settings demands forming abstract representations, integrating information across the document in addition to local representation, and generating coherent answers, which may not only be a span of text @xcite . due to requiring unique and multidimensional skills, narrative question answering has become an important task to assess and enhance the various complex reading comprehension abilities of both humans and machines @xcite @xcite . however, a consensus is lacking on the taxonomy of skills that narrative question answering represents and is suitable for assessment @xcite . task-level skill definitions only focus on one characteristic of the task, such as format, and ignore the multidimensional aspect of narrative question answering. in contrast, question-level skill definitions focus on identifying fine-grained skill definitions. previously, several question-level skill taxonomies have been proposed. some of these taxonomies (e.g., @xcite concentrate solely on challenging reading comprehension skills, such as commonsense reasoning, omitting others. alternatively, other taxonomies (e.g., @xcite group skills in a manner that a question can be attributed to several skills within the same dimension, e.g., the question can be associated with both temporal and causal skills from reasoning skill dimension, creating challenges during skill evaluation. recent narrative reading comprehension taxonomies do not pose these concerns and incorporate clearly distinguishable skills within skill dimensions @xcite . however, both of the taxonomies focus on a few skill dimensions. additionally, the taxonomy of @xcite does not clearly define the explicit and implicit skills; thus, adopting this taxonomy might lead to confusion in skill evaluation or in developing methods based on these skill dimensions (e.g., @xcite . we conclude that the current literature lacks a comprehensive narrative question answering skill taxonomy that accurately defines relevant skills without omitting key skills or skill dimensions. to address the identified problems, we introduce a taxonomy that synthesizes skills and skill dimensions from @xcite and @xcite , provides accurate definitions for implicit and explicit questions while introducing answer generation skills. our taxonomy is structured around four skill dimensions: narrative elements, representation scope, knowledge gap filling, and generation. these skill dimensions encompass both understanding and answering skills. we start by reviewing the previously proposed skill taxonomies and discuss their limitations in section 2. we introduce our skill taxonomy in section 3 and consider the skill assessment and learning opportunities of the proposed taxonomy in section 4. ## our skill taxonomy our skill taxonomy combines elements from prior narrative reading comprehension taxonomies, provides a redefined perspective on implicitness (referred to as knowledge gap filling), and introduces a generation skill dimension. we categorize the skill dimensions into two parts: ## skill assessment and learning effective and fair skill evaluations rely heavily on precisely defining and annotating the dimensions and features under study. in past instances, conducting focused assessments and fair evaluations posed challenges due to combining multiple skill dimensions into one and overlapping numerous skill features. our taxonomy outlined in section 3 addresses these issues by distinguishing each skill dimension and its associated features. annotating existing datasets and constructing future datasets based on our taxonomy will assist in conducting focused assessments and ensuring fair skill evaluations. the distinguishability of skill dimensions allows the study of individual dimensions in isolation for focused assessment. for fair skill evaluation, we assert that every skill feature within each dimension holds equal significance, and a single skill narrative: in terms of skill learning, each skill dimension becomes a focal point for improvement. previously, methods have been developed for enhancing input representations with narrative elements (e.g., @xcite , utilizing knowledge-based methods to enhance implicitness (e.g., @xcite , and employing pointer generator networks @xcite to improve the generation ability of models (e.g., @xcite @xcite . it is crucial to note that some of these previous efforts (e.g., @xcite have relied on inaccurate definitions of skills. we anticipate enhanced performance in narrative question answering by considering model development and annotation based on accurate skills definitions. furthermore, existing skill-based methods often concentrate on improving a single dimension or even a specific skill. we argue that there is con-siderable room for improvement by shifting the focus to multiple skill dimensions during method development. we believe assigning equal significance to each skill feature throughout the development process is key to achieving more robust and effective results. ## conclusion narrative question answering is a crucial task to assess and enhance complex reading comprehension skills. however, there is no consensus regarding the classification of skills that narrative question answering entails and whether they are suitable for evaluation. the current research lacks a comprehensive taxonomy that contains and correctly defines relevant essential skills and skill dimensions. in this paper, we propose a skill taxonomy for narrative question answering to address these challenges. our taxonomy synthesizes and redefines skills from existing taxonomies while incorporating a generation skill dimension. our taxonomy contains distinguishable skills within narrative elements, representation scope, knowledge gap filling, and generation skill dimensions. we hope that our taxonomy will facilitate focused and fair multidimensional skills assessment of narrative question answering and motivate further development of skill-learning methods.