SEMQA : Semi-Extractive Multi-Source Question Answering (2024)

Abstract: Recently proposed long-form question answering (QA) systems, supported by large language models (LLMs), have shown promising capabilities. Yet, attributing and verifying their generated abstractive answers can be difficult, and automatically evaluating their accuracy remains an ongoing challenge.In this work, we introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. Specifically, Semi-extractive Multi-source QA (SEMQA) requires models to output a comprehensive answer, while mixing factual quoted spans—copied verbatim from given input sources—and non-factual free-text connectors that glue these spans together into a single cohesive passage. This setting bridges the gap between the outputs of well-grounded but constrained extractive QA systems and more fluent but harder to attribute fully abstractive answers. Particularly, it enables a new mode for language models that leverages their advanced language generation capabilities, while also producing fine in-line attributions by-design that are easy to verify, interpret, and evaluate. To study this task, we create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions, and define text-based evaluation metrics. Experimenting with several LLMs in various settings, we find this task to be surprisingly challenging, demonstrating the importance of QuoteSum for developing and studying such consolidation capabilities.

## introduction large language models (llms) have recently demonstrated impressive capabilities across nlp tasks @xcite and in particular, in answering general questions @xcite . as a result, an increasing number of users interact with such models when seeking information. although llms often return high-quality and correct results, they can still make convincing-sounding mistakes (e.g., relying on false or out-dated information @xcite @xcite ). added citations can help the reader verify the generated answers @xcite , but might be inaccurate themselves and add a false impression of reliability @xcite . furthermore, evaluating fully-abstrative long-form qa systems remains a challenge, mostly involving either running another large qa model or expensive human ratings. in this work, we introduce the task of semiextractive multi-source qa (semqa) for answering multi-answer questions. specifically, given a question and a set of retrieved passages, the goal is to generate a summarized and well-grounded answer that interleaves verbatim extracted spans of factual statements with free-text connectors. including explicitly extracted spans in long generated answers provides multiple advantages. for example, it readily supports fast and easy verification by the reader-only requiring to confirm the context in the source. also, it simplifies and standardizes the evaluation process. instead of relying on model-based metrics that are expensive to run, hard to interpret, and depend on the performance of the measuring model @xcite , extracted spans can be readily evaluated for both recall and precision with standard string-matching metrics @xcite . finally, explicitly quoting spans can simplify the expensive auto-regressive generation process of lms, enabling potential efficiency gains @xcite . consider the example in figure 1 that asks when was "i'll be home for christmas" released. a fullyextractive format would likely only return the set of short-answers without any contextualization (e.g., the song "i'll be home for christmas" was originally released by bing crosby in 1943. the title would later be used in media like the film i'll be home for christmas, which was released on november 13, 1998. i'll be home for christmas is also a christmas-themed extended play released by epic records and syco entertainment on november 24, 2014. additionally, an album with the same title was released on october 7, 2008. quoted answer: question: figure 1: in semqa, the answer combines information from multiple sources while explicitly extracting factual spans (highlighted in different colors per source), and connecting them into a coherent well-grounded passage. to study the semqa task, we construct a dataset of questions, relevant passages, and humanwritten semi-extractive answers. our dataset, named quotesum, includes both natural multisource questions @xcite and generated multi-answer questions @xcite . we created an annotation pipeline where writers are asked to compile answers while using a tool that visualizes and keeps track of copied spans. in the task definition, we ask writers to use the provided quoting mechanism for all factual statements, while ensuring fluency of the text. writers can choose to skip certain sources if they don't include additional helpful content and cite only a subset of the given sources, simulating possible noise in the retriever module (see §4). we choose to focus on the multi-source setting since we argue it to more clearly benefit from this style. however, our approach can be easily relaxed to single-source and single-answer questions. by focusing on multi-answer questions we also emphasize the main challenges of this task, requiring models to (1) correctly find an answer in each source; we use quotesum to fine-tune large qa models and evaluate both supervised and in-context llms. we find llms to obtain remarkable performance with only a few quotesum examples. however, quotesum-tuned models perform best, indicating the usefulness of our data. we also run a user study comparing semqa answers to cited abstractive answers @xcite , and find semqa answers to be more comprehensive and much easier to be manually verified by readers. overall, our experiments reveal that the semqa task is surprisingly challenging even for modern llms. we define text-based metrics for semqa, creating easy-to-compute model-free measures to track the progress of models towards solving this task, capturing the fluency, preciseness and comprehensiveness of the generated quoted answers. our main contributions include: ## related work several open-domain qa datasets use short extracted spans or entities as target answers @xcite @xcite @xcite . the recent advancements in llms allowed expanding automatic qa systems to more challenging settings such as long-form answers @xcite , answers with multiple entities @xcite , and ambiguous questions @xcite . while most recent llm-based qa models focus on abstractive free-text closed-book or retrieve-then-read set-tings @xcite , evaluating these models remains a challenge @xcite , even when answers have citations @xcite @xcite @xcite @xcite -mostly involving expensive model-based measures @xcite . similar to our motivation to improve the faithfulness of generated answers, @xcite suggest an extract-and-decontextualize pipeline that starts with a fully extractive long-form answer, and edits it for improving fluency. however, they focus on single-source questions which don't require multi-answer consolidation. regarding evaluation, recent studies identified some limitations of lexical metrics for general qa evaluation @xcite . however, our task formulation that limits the output space to the given sources, combined with the semiextractive nature and collecting multiple references, allow us to define model-free evaluation metrics that overcome most of the discussed limitations (see §3). ultimately, a practitioner might choose to combine multiple different evaluators. ## semi-extractive multi-source qa to formally define the semqa setting, we assume an input question q and a set of two or more passages p that possibly include an answer to question q. to simplify the setting, we assume here that the passages @xmath0. our goal is to generate an answer (q, p) → a that (1) answers q, covering all components and aspects based on information from p, (2) is as extractive as possible with respect to p (explicitly marking extractions and their source), and (3) is concise foot_1 and fluent. there is a natural trade-off between the last two requirements for a. this balance is necessary since naïvely maximizing only (1) and (2) would lead to simply concatenating all passages in p. in contrast, maximizing only (3) would lead to an empty answer. we resolve this tension by seeking to ensure extractiveness for entities and for core factual statements, and to prefer fluency otherwise. by explicit quoting marks, we mean that each extracted span should be marked as such, with an indicator for which passage it was taken from. the passage indicator is important for distinguishing identical spans across passages (e.g., "i'll be home for christmas" in figure 1 ) and connecting each statement in the answer with the original supporting source. for brevity, we refer to such explicitly marked semi-extractive answers as quoted answers, even though the answer is not required to include textual quotation marks around quoted spans. we note that some questions might not be adequately answered with this format. moreover, the feasibility of composing a sufficient answer also depends on the given passages. we argue (and demonstrate), though, that many questions can be answered in a semi-extractive format. this format provides several key advantages described below. verifiability. it is generally easier to verify answers that directly quotes factual statements compared to verifying answers that paraphrase information, even when sentences are attributed to supporting sources. in the latter case, the reader would need to decompose the generated text into statements, review the sources, locate relevant information pieces, and compare the generated statements against them. in contrast, when factual statements are extracted verbatim, verifying the generated text broadly simplifies to ensuring that the statements weren't taken out of context. the benefits of quoted spans are even more pronounced in sentences that aggregate information from multiple sources. rather than citing many sources, leading to combinatorial growth, attributing at a span-level drastically simplifies the verification task. in §5.1, we empirically show that this hypothesis holds with human readers. ## the quotesum dataset to study the semqa task, we create a new multianswer qa dataset with (q, p, a) triplets, where a is a human-written semi-extractive answer. we call this dataset quotesum, as it includes quoted answers to the input questions, based on the provided sources. we use square brackets and the index of the respective source to explicitly mark the extracted spans. for example, the first sentence of the answer in figure 1 would be: the song "[ 1 i'll be home for christmas ]" was originally released by [ 1 bing crosby ] in @xcite to mark the spans extracted from source 1. throughout this paper, we replace the textual marks with colored highlights for presentation. next, in §4.1 we describe our process for collecting (q, p) pairs based on the paq @xcite and nq @xcite datasets. then, in §4.2 we detail our crowd-sourcing task for writing quoted answers. ## experimental evaluation we study the performance of both fine-tuned lms, and in-context few-shot llms, as described below. we follow the evaluation metrics described in §3.1, and also run a user study as detailed in §5.1. fine-tuned models. we fine-tune different sizes of t5 models @xcite on the quotesum training set. we also fine-tune flan instructiontuned t5 models @xcite . see appendix d for more details. in-context learning. we evaluate variants of few-shot palm2 @xcite . we construct a dynamic prompt that collects examples for each test question q i by retrieving the most similar questions from the quotesum training set according to sentence-t5 @xcite embedding cosine similarity (qsum). we also experiment with the alce @xcite prompt that uses fullyabstractive answers with in-line citations (e.g., @xcite to cite source 1), and with qsum-s prompts that convert qsum examples to sentence-level citations. see appendix d for details and example prompts. ## conclusion we introduce the task of semi-extractive multisource qa (semqa), and a new comprehensive dataset with human-written semqa examples (quotesum) with text-based evaluation metrics. our experiments and analysis reveal the benefits of semqa for enabling llms to generate wellgrounded answers that are fast to verify against external sources, and easy to interpret and evaluate against references-avoiding the reliance on model-based measures. we hope that semqa and the released data will facilitate further research on improving llm capabilities around reliable qa. we make the quotesum dataset and implementation of the semqa metrics available at: https://github.com/ google-research-datasets/quotesum .