Read before Generate! Faithful Long Form Question Answering with Machine Reading (2022)

Abstract: Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. While current work on LFQA using large pre-trained model for generation are effective at producing fluent and somewhat relevant content, one primary challenge lies in how to generate a faithful answer that has less hallucinated content. We propose a new end-to-end framework that jointly models answer generation and machine reading. The key idea is to augment the generation model with fine-grained, answer-related salient information which can be viewed as an emphasis on faithful facts. State-of-the-art results on two LFQA datasets, ELI5 and MS MARCO, demonstrate the effectiveness of our method, in comparison with strong baselines on automatic and human evaluation metrics. A detailed analysis further proves the competency of our methods in generating fluent, relevant, and more faithful answers.

## introduction long-form question answering (lfqa) is a task to generate an in-depth, paragraph-length answer for a given question @xcite . it is important since many of the everyday questions that humans deal with and pose to search engines require multisentence explanations @xcite (e.g. why/how..?). it can be integrated with a search engine @xcite , or a virtual conversation agent, and can also be used to generate explanations as a complement to short-phrase answers for open-domain questions @xcite , or to answer open-ended questions like those from reddit forum "explain like i'm five" @xcite . lfqa is quite a challenging task. it often involves searching a large external knowledge source figure 1 : an example from ms marco @xcite dataset. we highlight the unfaithful snippets from other model. our model(rbg) generate more factually accurate answer. while recent work on lfqa @xcite focuses primarily on the first challenge, and has produced fluent and somewhat relevant content, the latter faithfulness challenge has not been explored. however, the faithfulness issue is quite important for lfqa. as the example in fig. 1 shown, a fluent and relevant but unfaithful answer (high-light in green) will mislead users. in this paper, we propose a novel end-to-end framework named rbg (read before generate) for lfqa to address the aforementioned challenges. the key idea for enhancing answer faithfulness is to augment the generation process with predicted salient information which can be viewed as an emphasis on answer-related facts. specifically, we combine a seq2seq language modelbased generator with a machine reading comprehension (reader) module. the reader produces an evidence probability score for each sentence, which will be integrated with the generator for final distribution prediction. we perform evidence fusion in a similar way as fid @xcite to equip the pre-trained language model with multiple input documents for generation. to further enhance the factual grounding ability of rbg, we propose an additional pre-training task to encourage the model to rely more on retrieved documents to generate factual statements. the details are explained in section 2. we conduct thorough experiments on our method and several baselines on eli5 @xcite , the only publicly available large-scale lfqa dataset, and also on ms marco @xcite passage ranking data, which can also be transformed into an answer generation task. the proposed method tops the public leaderboard of the kilt @xcite benchmark on the eli5 dataset. it also outperforms the baselines, including non-retrieval and retrieval-based methods, such as dpr-bart @xcite , rag (izacard and grave, 2021) and fid @xcite , with improvement on the automatic evaluation results on the ms marco dataset. human evaluation results further validate that our proposed framework can improve the generation quality in terms of relevance and factual correctness. our contributions are summarized as follows: ## a state-of-the-art lfqa system to generate in-depth, long-form answers for a given general domain question, we first use a retriever to search for relevant information from a large external knowledge source. then our reader and the generation module take the multiple retrieved documents together with the question as input to generate the answer. specifically, the reader module adopts a machine reading comprehension (mrc) model to produce an evidence score for each sentence in each document, while the generator, which adopts a large pre-trained seq2seq language model, fuses the sentence evidence score into its generation process. our framework is shown in figure 2 . ## further analysis we conduct further analysis on the results, considering that lfqa is a complicated but less explored task, which deserves a complete investigation. ## conclusion we propose a new end-to-end framework rbg that jointly models answer generation and machine reading to tackle the faithfulness issue in lfqa. experiments on two lfqa datasets, eli5 and ms marco, demonstrate the effectiveness of our method in comparison with strong baselines on automatic and human evaluation metrics. the detailed analysis further proves the competency of our method in generating fluent, relevant, and more faithful answers. we also propose to evaluate the factual correctness of lfqa model by answering questions of extractive qa tasks (e.g., natural questions), which may be helpful to evaluate the faithfulness of lfqa model efficiently.