Frustratingly Hard Evidence Retrieval for QA Over Books (2020)

Abstract: A lot of progress has been made to improve question answering (QA) in recent years, but the special problem of QA over narrative book stories has not been explored in-depth. We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We further investigate how state-of-the-art open-domain QA approaches can help BookQA. Besides achieving state-of-the-art on the NarrativeQA benchmark, our study also reveals the difficulty of evidence retrieval in books with a wealth of experiments and analysis - which necessitates future effort on novel solutions for evidence retrieval in BookQA.

## introduction the task of question answering has benefited largely from the advancements in deep learning, especially from the pre-trained language models(lm) @xcite . while question answering over single passage (reading comprehension datasets) and over the large-scale open-domain corpora (open-domain qa) have largely benefited from these, the performance of qa over book stories (bookqa) lags behind. for example, the most representative benchmark in this direction, the narrativeqa @xcite which was released three years ago -the current state-of-the-art methods only show marginal improvement over the first baselines. there are several challenges in narrativeqa which slow down the research progress. first, the narrative stories lead to a new writing style which differs from previous works over formal texts like wikipedia. second, the long inputs of books are beyond the processing ability of neural models so evidence identification from a whole book is critical. third, narrativeqa is a generative task, and many of the answers cannot be exactly matched in the original books. hence, the generative qa models are required. finally and most importantly, the dataset does not provide annotations of the supporting evidence. while this makes it a realistic setting like open-domain qa, together with the generative nature of the answers, also makes it difficult to infer the supporting evidence similar to most of the extractive open-domain qa tasks. the requirements around evidence identification and the missing supporting evidence annotation make bookqa task similar to open-domain qa. in this paper, we first study whether the ideas used in state-of-the-art open-domain qa systems can be extended to improve bookqa including: (1) the neural ranker-reader pipeline @xcite , where a neural ranker is used to select related passages (evidence) given a question from a large candidate sets; (2) the usage of pre-trained lms as reader and ranker, such as gpt @xcite , bert @xcite and their followup work; (3) the distantly supervised and unsupervised training techniques @xcite @xcite @xcite ) that help rankers learn more from noisy gold data. by training a ranker-reader framework on bookqa, we successfully achieve a new state-ofthe-art on narrativeqa using both generative and extractive readers. based on these results and our analysis, we observe the followings: ## conclusion we explored the bookqa task and systemically tested on narrativeqa dataset different types of models and techniques from open-domain qa. our proposed approaches bring significant improvements to the state-of-the-art across different metrics. our insight and analysis lay the path for excit-ing future work in this domain.