Concise Answers to Complex Questions: Summarization of Long-form Answers (2023)

Abstract: Long-form question answering systems provide rich information by presenting paragraph-level answers, often containing optional background or auxiliary information. While such comprehensive answers are helpful, not all information is required to answer the question (e.g. users with domain knowledge do not need an explanation of background). Can we provide a concise version of the answer by summarizing it, while still addressing the question? We conduct a user study on summarized answers generated from state-of-the-art models and our newly proposed extract-and-decontextualize approach. We find a large proportion of long-form answers (over 90%) in the ELI5 domain can be adequately summarized by at least one system, while complex and implicit answers are challenging to compress. We observe that decontextualization improves the quality of the extractive summary, exemplifying its potential in the summarization task. To promote future work, we provide an extractive summarization dataset covering 1K long-form answers and our user study annotations. Together, we present the first study on summarizing long-form answers, taking a step forward for QA agents that can provide answers at multiple granularities.

## introduction long-form answers @xcite , as compared to span-based short answers @xcite , can provide comprehensive answers to a broader set of questions @xcite . while providing comprehensive information in multiple sentences is helpful, users often prefer short and concise answers to their questions when possible @xcite . today's search engines already present concise answers by highlighting the most relevant parts from the passage excerpts. in this paper, we present the first study on summarizing long-form answers. summarizing long-form answers introduces a new challenge in addition to the faithfulness and fluency challenges of generic summarization which mostly focus on news articles @xcite : the summary output should still provide a reasonable answer to the original question. we take inspiration from a recent study @xcite that reports that up to 40% of sentences in long-form answers contain non-essential information, such as providing background information or examples @xcite , which demonstrates the potential for compressing long-form answer. we first aim for an extractive summarization model and collect sentence-level annotations on long-form answers, where annotators identify sentences that address the question directly and can serve as the "summary". 1 we collect a dataset covering 1,134 examples, each consisting of a question, a long-form answer, and a set of summary sentences. to improve the extractive summaries collected, we propose a simple and yet novel summarization approach, extract-and-decontextualize, which first extracts summary sentences and rewrites them to stand-alone @xcite . compared to abstractive summarization models trained on noisy distantly supervised datasets (e.g. cnn/dm @xcite and xsum @xcite ) which encourage paraphrasing but also hallucinations @xcite @xcite , decontextualization makes minimal edits to the original sentence, preserving its meaning while improving its fluency. how well do summarization approaches perform in this new domain -can generated summaries provide fluent, adequate answers to the questions, while preserving the semantics of the original longform answers? we evaluate fine-tuned abstrac- input system summarized answer adequacy faithful q: why does car sickness seem to hit the hardest when you look down at your phone, book, etc.? a: the brain perceived motion because it receives information from the eyes, ears, and muscles. when these parts send conflicting information, the brain doesn't know which is right and which is wrong, and this is what causes motion sickness. an example of this is when reading a book while you are in a moving car. to your eyes, the book is stationary while your inner ear and the rest of your body can feel a sense of motion. this would likely cause car sickness. abstractive the brain gets confused when it receives conflicting information about motion from different parts of the body, and this can cause car sickness. yes yes gold extractive when these parts send conflicting information, the brain doesn't know which is right and which is wrong, and this is what causes motion sickness. partially yes decontext when -these parts +the eyes, ears, and muscles send conflicting information, the brain doesn't know which is right and which is wrong, and this is what causes motion sickness. yes yes q: how come obama during his supermajority in both houses wasn't able to pass any legislation he wanted? a: 1) senators are normally seated in january. [. . . ]al franken (who would've been #60) was not seated until july 7.2) ted kennedy was dying and had not cast a vote since[. . . ]note that sept 24-feb 4 is about 20 working days, due to recess and holidays.3) so, for about 20 working days, the senate democrats could have broken a filibuster if you could get every single one of them to agree on something. [. . . ] this did not go well. abstractive the senate democrats were unable to pass any legislation during obama's supermajority due to a lack of 60 votes needed to break a filibuster, due to al franken not being seated until july 7 and ted kennedy's death in august 2009. partially yes gold extractive so, for about 20 working days, the senate democrats could have broken a filibuster if you could get every single one of them to agree on something. no no / decontext ## background and motivation the focus of our study is to find a concise answer to a complex question @xcite . one way to generate a concise answer is through controllable generation, where the long-form qa model is instructed to generate an answer given a pre-specified length. however, long-form question answering remains challenging, both in terms of modeling @xcite and reliable evaluation @xcite . existing models often hallucinate @xcite even when paired with relevant evidence documents. instead of generating a concise answer from scratch, we summarize an existing long-form answer, leveraging a large amount of user-written long-form answers often in community-driven qa forums like eli5 in reddit. how feasible would it be to summarize existing long-form answers? @xcite conducted an in-depth study on the structure of such longform answers, assigning one of six functional roles (answer, answer summary, organizational sentence, auxiliary information, and example) to each sentence in long-form answer. the study suggests sentences corresponding to "answer summary" captures the salient information and "often suffice by themselves as the answer to the question." furthermore, they suggest up to 40% of sentences belongs to roles (e.g., auxiliary information) that are not necessary to answer the question, suggesting summarizing existing answer is viable. we follow their study and collect larger-scale data focusing on the "answer summary" role to study the summarization of long-form answers. summarizing existing answers will support providing a consistent answer set of different granularities, where the users can expand condensed answer to see a more detailed version of the same answer. consistent answers at multiple granularities are harder to enforce with a controllable generation approach. for instance, if we generate a five-sentence answer from the raw evidence set, the five-sentence answer can contain information absent in the tensentence answer. lastly, retrieval-augmented long-form qa models @xcite resemble query-focused summarization. query-focused summarization @xcite often studies challenging multi-document settings, where the input text is summarized focusing on a particular query, provided at inference time content control. a difference to our setting is that a long-form answer is written for the question q, presenting already synthesized information tailored for the question. foot_1 ## extractive summary for long-form answers we first introduce our annotation task of identifying key sentences for long-form answers, which will be used as an extractive summary. extractive summaries allow easier data collection and evaluation but can suffer from disfluency and incoherence. thus, we manually evaluate our collected gold extractive summaries in section 5.