Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study (2021)

Abstract: Recent advancements in open-domain question answering (ODQA), that is, finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a ∼7% absolute improvement on ROUGE-L. (2) We further analyze the detailed challenges in Book QA through human studies.1 Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.

## introduction recent question-answering (qa) models have achieved or even surpassed human performance on many challenging tasks, including singlepassage qa 2 and open-domain qa (odqa). 3 nevertheless, understanding rich context beyond text pattern matching remains unsolved, espe- * equal contribution. xm built the whole system, implemented the data preprocessing pipeline, hard em ranker, and all the reader modules, and conducted all the qa experiments. cy implemented the unsupervised ict ranker and the first working version of fid, and was responsible for the final ranker module. my is the corresponding author, who proposed and led this project, built the ranker code base (until the ds ranker), designed the question schema and conducted its related experiments and analysis in part ii. despite its similarity to standard odqa tasks, foot_0 that is, both requiring finding evidence paragraphs for inferring answers, the book qa has certain unique challenges @xcite : (1) the narrative writing style of book stories differs from the formal texts in wikipedia and news, which demands a deeper understanding capability. the flexible writing styles from different genres and authors make the challenge severe; (2) the passages that depict related book plots and characters share more semantic similarities than the wikipedia articles, which increases confusion in finding the correct evidence to answer a question; second, we conduct human studies to quantify the challenges in book qa. to this end, we design a new question categorization schema based on the types of reading comprehension or reasoning skills required to provide the correct answers. precisely, we first define the basic semantic units, such as entities, event structures in the questions and answers. the question category thus determines the types of units and the relations between the units. we annotate 1,000 questions accordingly and discover the significantly distinctive statistics of the narrativeqa dataset from the other qa datasets, mainly regarding the focus of event arguments and relations between events. we further give performance decomposition of our system over the question categories, to show the detailed types of challenges in a quantitative way. in summary, our comprehensive study not only improves the state-of-the-art with careful utiliza-tion of recent odqa advancements, but also reveals the unique challenges in book qa with quantitative measurements. book qa previous works @xcite @xcite also adopt a ranker-reader pipeline. however, they have not fully investigated the state-of-the-art odqa techniques. first, the narrativeqa is a generative qa task by nature, yet the application of the latest pre-trained lms for generation purposes, such as bart, is not well-studied. second, lack of fine-grained supervision on evidence prevents earlier methods from training a neural ranking model, thus they only use simple bm25 @xcite based retrievers. an exception is @xcite , who construct pseudo distance supervision signals for ranker training. another relevant work @xcite uses book summaries as an additional resource to train rankers. however, this is different from the aim of the book qa task in answering questions solely from books, since in a general scenario the book summary cannot answer all questions about the book. our work is the first to investigate and compare improved training algorithms for rankers and readers in book qa. ## analysis part i: experimental study this section describes our efforts of applying or adapting the latest open-domain qa ideas to improve book qa ranker/reader models. figure 3 summarizes our inspected approaches. the experimental results quantify the challenges in book qa beyond open-domain qa. ## evaluation part i: qa system ablation we evaluate the overall book qa system, and the individual modules on narrativeqa. implementation details: for rankers, we initialize with bert-base-uncased. for readers, we use bert-base-uncased, gpt2-medium, bart-large, and t5-base. the readers use top-3 retrieved passages as inputs, except for the fid reader which uses top-10, making the readers have comparable time and space complexities. ## evaluation part ii: qa system performance decomposition most difficult question types: the performance breakdown shows that all three event-relation types (causal, temporal, and nested) are challenging to our qa systems. the causal relation is the most difficult type with the lowest qa performance. the result confirms that the unique challenge in understanding event relations is still far from being well-handled by current machine comprehension techniques, even with powerful pre-trained lms. moreover, these types can also be potentially improved by the idea of complementary evidence retrieval @xcite @xcite in odqa. besides the three event-relation types, the event -attribute and event -triggers are also challenging to the extractive system, because the answers are usually long textual mentions of events or states that are not extractable from the passages. challenging types for the reader: by checking the performance gaps of the generative system and the ranker, we can tell which types are difficult mainly for the reader. 11 the event -concept type poses more challenges to the reader, given that the ranker can perform well on them but the overall qa performance is low. these questions are challenging mainly due to the current readers' difficulty in understanding the event structures, since their answers are usually extractable from texts. breakdown onto answer types: to better understand the challenges of non-extractable answers, we show the performance on each answer type in ## conclusion we conduct a comprehensive analysis on the book qa task, taking the representative narrativeqa dataset as an example. firstly, we design the book 1042 qa techniques by borrowing the wisdom from the cutting-edge open-domain qa research and demonstrate through extensive experiments that (1) evidence retrieval in book qa is difficult even with the state-of-the-art pre-trained lms, due to the factors of rich writing style, recurrent book plots and characters, and the requirement of high-level story understanding; (2) our proposed approaches that adapt pre-trained lms to books, especially the prereading technique for the reader training, are consistently helpful. secondly, we perform a human study and find that (1) a majority of questions in book qa requires understanding and differentiating events and their relations; (2) the existing pre-trained lms are deficient in extracting the inter-and intra-structures of the events in the book qa. such facts lead us towards the event understanding task for future improvement over the book qa task.