Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2022)

Abstract: Retrieval-augmented generation models have shown state-of-the-art performance across many knowledge-intensive NLP tasks such as open-domain question answering and fact verification. These models are trained to generate a final output given retrieved passages that can be irrelevant to an input query, leading to learning spurious cues or memorization. This work introduces a method to incorporate evidentiality of passages—whether a passage contains correct evidence to support the output—into training the generator. We introduce a multi-task learning framework to jointly generate the final output and predict the evidentiality of each passage. Furthermore, we introduce a new task-agnostic method for obtaining high-quality silver evidentiality labels, addressing the issues of gold evidentiality labels being unavailable in most domains. Our experiments on five datasets across three knowledge-intensive tasks show that our new evidentiality-guided generator significantly outperforms its direct counterpart on all of them, and advances the state of the art on three of them. Our analysis shows that multi-task learning and silver evidentiality mining play key roles. Our code is available at https://github.com/AkariAsai/evidentiality_qa

## introduction knowledge-intensive tasks, including open-domain question answering (qa) and fact verification, require evidence passages related to an input query to be retrieved from a large collection of passages (e.g., wikipedia). recently, most successful methods use retrieval-augmented generation @xcite , which is a pipeline approach of first training a retriever model @xcite for retrieving passages and then independently training a generator model @xcite given the passages. the fate of the furious the fate of the furious was released in the united states on april 14, 2017. ## experimental setups we experiment on three knowledge-intensive tasks: open-domain qa, fact verification, and knowledgeenhanced dialogue. statistics for each dataset are provided in ## results and analysis our approach significantly improves over its direct counterpart on all datasets, and outperforms all prior published results on faviq-a, fever and wow, advancing their state-of-the-art performance. ## conclusion augmenting pre-trained generation models with retrievers has shown to be effective in many knowledge-intensive tasks; however, they often rely on spurious cues or generate hallucinations during inference. we introduce a multi-task learning objective the combines answer generation and evidentiality prediction. we propose task-agnostic data mining techniques to obtain silver evidentiality labels to enable this auxiliary training. our experiments across five datasets show large performance improvements over baselines and our evidentialityguided generator advances the state-of-the-art performance on faviq-ambig, fever and wow. our analysis shows that multi-task learning and silver evidentiality mining both contribute to the performance improvements by helping the model learn to focus on and generate answers from more relevant passages.