Retrieving Contextual Information for Long-Form Question Answering using Weak Supervision (2024)

Abstract: Long-form question answering (LFQA) aims at generating in-depth answers to end-user questions, providing relevant information beyond the direct answer. However, existing retrievers are typically optimized towards information that directly targets the question, missing out on such contextual information. Furthermore, there is a lack of training data for relevant context. To this end, we propose and compare different weak supervision techniques to optimize retrieval for contextual information. Experiments demonstrate improvements on the end-to-end QA performance on ASQA, a dataset for long-form question answering. Importantly, as more contextual information is retrieved, we improve the relevant page recall for LFQA by 14.7% and the groundedness of generated long-form answers by 12.5%. Finally, we show that long-form answers often anticipate likely follow-up questions, via experiments on a conversational QA dataset.

## introduction the goal of long-form question answering (lfqa) is to provide in-depth answers to end-user questions @xcite . for example, for the user question "when did lionel messi start his career?" a direct answer would be: @xcite which is the date of his first team debut. however, this answer naturally sparks a series of follow-up questions to obtain more contextual details @xcite : "for which club?" "in which match?" "what about his la liga debut?" "how did his career develop?" * work was done during an internship at amazon agi. a long-form answer aims to proactively supply a more complete and detailed response beyond the succinct direct answer, essentially anticipating such follow-up questions: lionel messi started his career as a professional football player with fc barcelona. he made his first-team debut in a friendly against porto on 16 november 2003, at the age of 16. . . . we thus draw parallels between lfqa and conversational question answering (convqa) @xcite @xcite @xcite and hypothesize that they can be treated as complementary tasks. existing approaches for lfqa based on retrieval-augmented generation @xcite @xcite typically utilize retrievers that are optimized to obtain direct answers to questions. such retrieval systems thus often fail to retrieve relevant context, which is needed to generate faithful and comprehensive long-form answers. approach. we propose to train a specialised retriever for the task of lfqa that not only retrieves direct answers for a question, but also retrieves additional context required for grounding long-form answers. our goal is to retrieve both direct answers and contextual information in one shot. the major bottleneck for training a retriever for lfqa is the absence of training data. lfqa datasets @xcite contain questions and long-form answers but not the ground-truth passages required to produce those long-form answers. in this work, we propose a mechanism to automatically infer silver passages, designed to ground both (i) the direct answers, and (ii) the contextual information. these passages should provide sufficient evidence to support information in the long-form answer. this is different from previous work on factoid short-form qa, which identified such passages by matching only against the direct answers @xcite . based on these silver passages, we train bertbased re-ranking models @xcite . these re-rankers are applied to the initial retrieval results, to enhance recall of contextual information in the top-ranked passages. these top-ranked passages are then provided as input to an llm to generate the long-form answer. we conduct experiments on asqa @xcite , a dataset for lfqa, and show that we substantially improve end-to-end qa performance, while also increasing the groundedness of the longform answer w.r.t. the retrieved passages. experiments on the convqa dataset con-vmix @xcite demonstrate that our method generates long-form answers that often also contain answers to the follow-up questions, when provided only with the first question of the conversation as input. this indicates that lfqa can indeed anticipate likely follow-up questions. ## identifying silver passages the key idea for obtaining such silver passages is to utilize both the long-form answers (lfas) and direct answers (das), as annotated in the asqa dataset, jointly as a weak supervision signal for passage relevance. we retrieve a large set of passages first (say 100), using first-stage retrieval @xcite , and then choose up to k silver passages from this candidate pool. we considered three techniques for matching candidate passages against an lfa: (i) lexical matching, (ii) semantic similarity, and (iii) llm perplexity. our proposed approach matches against a combination of both lfas and das, and we also compare these against matching only with das. lexical matching with lfa. we initially evaluated (i) token recall, (ii) jaccard similarity between token sets, and (iii) rouge-l @xcite , and found that plain token recall works best. we thus compute the matching score for a pair of a candidate passage p and the lf a as: llm perplexity of lfa. inspired by the approach used in toolformer @xcite , we compute the llm perplexity of the target lf a (of length n), given a candidate passage p, as follows: pseudo-code for silver approach. algorithm 1 in the appendix illustrates the end-to-end workflow of deriving silver passages, fine-tuning the re-ranker, and fine-tuning the llm on lfqa data. ## experiments benchmarks. we conduct experiments on two datasets: (i) asqa @xcite , a dataset for lfqa, and (ii) convmix @xcite , a dataset for convqa. we use asqa for fine-tuning the re-rankers and llms. experiments on convmix use the same models, thus also test the generalizability of the approach. retrieval metrics. for evaluating retrieval, we measure recall in the top-5 retrieved passages. on asqa, we compute direct answer recall as the fraction of das appearing in the retrieved passages. as a proxy for recall of contextual information, we measure wikipage recall as the fraction of the ground-truth relevant wikipages matched with our retrieval results. a wikipage is matched if we retrieve a passage from the respective page. on convmix, we measure direct answer recall / follow-up answer recall as the fraction of das for the first question / follow-up questions appearing in the retrieved passages. metrics. on asqa, we keep the metrics used in the original work, and use their evaluation code foot_1 . this includes rouge-l @xcite for measuring the overlap of the generated text with one of the two reference lfas. for answer correctness, the disambig-f1 (d-f1) metric is used, measuring the fraction of question interpretations answerable from the generated lfa. a pre-trained machine reading comprehension (mrc) @xcite model is used to this end. the dr metric combines the two metrics via the geometric mean. for convmix, inspired by the d-f1 metric, we compute c-f1 as the fraction of conversational questions answerable from the generated lfa. we use the same mrc model as for the d-f1 metric. in addition, we measure groundedness as the fraction of tokens in the answer that is also present in the retrieved passages. similar to lexical matching in sec. 2, stopwords are not considered. configuration. we use dpr @xcite foot_2 for first-stage retrieval, which was shown to outperform bm25 @xcite on asqa in previous work @xcite , and is still considered state-of-the-art on naturalquestions @xcite , which is a superset of asqa. we use vicuna 13b 1.5 @xcite as llm for generating lfqas and fine-tune it for 1 epoch using the long-form answers in the asqa dataset as the target output. the input to the model are the question and the top-5 passages retrieved either by dpr or by our silver re-rankers. we use the same llm prompt as in the original asqa paper to combine the question and the retrieved passages for consistency. further details on the setup in appendix a. ## related work long-form question answering. with the recent advances of llms @xcite @xcite question answering has evolved beyond crisp and direct answers @xcite @xcite toward supplying more in-depth and comprehensive passage-length responses. there has been exten- question "who played grandpa joe in charlie and the chocolate factory?" (from asqa) dpr + vicuna 13b willy wonka and the chocolate factory is a 1970 children's book by roald dahl. it has been adapted into various media, including a 1971 film and a 2005 film. in the 1971 film, grandpa joe is played by wilbur robinson. in the 2005 film, grandpa joe is played by david morris. -groundedness: 0.61 silver + vicuna 13b (proposed) charlie and the chocolate factory is a 2005 musical fantasy comedy film directed by tim burton and written by john august, based on the 1964 british novel of the same name by roald dahl. the film stars johnny depp as willy wonka and freddie highmore as charlie bucket. the storyline follows charlie, who wins a contest and, along with four other contest winners, is led by wonka on a tour of his chocolate factory, the most magnificent in the world. in the film, grandpa joe is played by david kelly. in the 2010 musical adaptation of charlie and the chocolate factory, grandpa joe is played by richard dreyfuss. -groundedness: 0.94 question "number of episodes in the sopranos?" (from convmix) dpr + vicuna 13b the sopranos is an american crime drama television series created by david chase. the series ran for 73 episodes over six seasons, from january 15, 1999, to june 11, 2007. -groundedness: 0.23 the sopranos is an american crime drama television series created by david chase. the series portrays the difficulties that fictional new jersey-based, italian american mobster tony soprano (james gandolfini) faces as he tries to balance his home life and his criminal organization. the series features tony's family members, mafia colleagues, and rivals in prominent roles and story arcs, most notably his wife carmela (edie falco) and protégé christopher moltisanti (michael imperioli). the series ran for 86 episodes over six seasons. -groundedness: 1.00 ## conclusion the retrieval of contextual information is often neglected in lfqa, while being an important ingredient for generating and grounding comprehensive long-form answers. we investigate techniques to obtain training samples providing such contextual information for training re-ranking models. we show that incorporating our re-rankers improves retrieval and qa performance on a lfqa dataset, yielding state-of-the-art performance on asqa. notably, our method enhances groundedness of generated texts by 12.5%, which can reduce factual hallucinations in answers. experiments on con-vmix show that our method, trained on asqa, is able to generalize to an unseen convqa dataset.