A nswer S umm: A Manually-Curated Dataset and Pipeline for Answer Summarization (2022)

Abstract: Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absence of a dataset to provide supervision for producing such summaries. Recent works propose heuristics to create such data, but these are often noisy and do not cover all answer perspectives present. This work introduces a novel dataset of 4,631 CQA threads for answer summarization curated by professional linguists. Our pipeline gathers annotations for all subtasks of answer summarization, including relevant answer sentence selection, grouping these sentences based on perspectives, summarizing each perspective, and producing an overall summary. We analyze and benchmark state-of-the-art models on these subtasks and introduce a novel unsupervised approach for multi-perspective data augmentation that boosts summarization performance according to automatic evaluation. Finally, we propose reinforcement learning rewards to improve factual consistency and answer coverage and analyze areas for improvement.

## introduction in a world of information overload and the ubiquity of discussion fora, there is a need for text summarization as a means of distilling relevant information into a concise form. the problem is even more pertinent for question answering within the context of community question answering (cqa) fora, where a person poses a question and can get an abundance of answers to sift through. ideally, an answer summary should cover the multiple perspectives found in the answers, where available. question: i recently relocated to usa and have no credit score. is secure credit card is the only option for me to start building my credit score? also please recommend which other credit cards are available for people like me to build credit score answer 1: if you have an amex from another country, you can get an amex in the us. american express has a separate system that is not as strongly country-dependent as, say, visa and mastercard... answer 2: secured credit cards are usually not very cost effective for building credit. find a local credit union, of medium to large size. a credit union is like a bank, but operates under slightly different rules, and is non-profit... answer 3: if you have had an american express card abroad, you can try and get a us amex... answer 4: if the country you came from has an hsbc, you can ask hsbc to use your credit rating from that country to give you an hsbc mastercard in the us... summary: there are a range of options available to you, although your chance of success will depend on the bank that you apply with. however, if you have previously had a card with hsbc or american express, the process may be simpler. other options could include borowing from a credit union or asking a friend or family member to be an additional cardholder with you. to date, most cqa fora have a notion of a 'best answer,' which is either manually chosen by the person who asked the question or by a moderator, or obtained via community ratings. work in this field typically makes use of this best answer as a proxy for summaries, i.e. the focus is on extractivelike summaries @xcite @xcite @xcite . datasets such as wikihowqa @xcite , which consists of a question, a long answer, and an answer summary, focus on answer selection and the summarization of a single answer. while cqasumm @xcite uses the chosen best answer as the answer summary, they also apply heuristics to ensure token overlap with the remaining answers. however, the best answer only presents one person's perspective and rarely captures the variety of perspectives discussed in a thread. furthermore, we find that the heuristics applied in cqasumm generally promote only long answers instead of multiple perspectives. to validate our hypothesis, we examine a set of 30 summaries from cqasumm and found that only 37% of the examples contained multi-perspective answers. in contrast, 75% of our dataset requires multi-perspective summaries. as alluded to above, although answer summarization is an important research topic with practical applications, there are no relevant datasets or techniques to address it effectively, i.e. no manually-curated dataset exists for the answer summarization problem, and no dataset decomposes the task into its constituent subtasks. this work tries to close the research gap in answer summarization; we develop an annotation pipeline for multiperspective abstractive answer summarization. we introduce the largest human-annotated dataset for answer summarization, containing components for sentence relevance, clustering, cluster summarization, and global answer summarization. we enlist ten professional linguists to contribute to our annotation efforts. we iterate over instructions and devise pre-pilot, pilot, and final annotation stages as well as re-annotation for quality assurance. we collect over 4,631 high-quality data points. for validation of our curated data set, we benchmark stateof-the-art models on the subtasks of this dataset and perform qualitative analysis to provide a clear baseline and directions for future work. we then propose a data augmentation pipeline to further boost summarization performance. to generate a silver multi-perspective summarization dataset, we introduce a pipeline for automatically creating multi-perspective bullet-point answer summaries for data augmentation, which boosts performance. we find that a strong baseline model trained on our human-annotated data inherently outputs factually consistent summaries, and model performance is improved by adding data from our automated pipeline. finally, we introduce entailment-based and semantic area rl rewards namely to analyze its effect on factual consistency and semantic coverage, ensuring we are capturing all factually relevant perspectives. foot_0 2 related work extractive answer summarization: much work has focused on the extractive summarization setting as an answer-ranking problem @xcite @xcite . @xcite find that only 48% of the best answers on yahoo! answers are unique best answers; there are multiple correct ways to answer a question. other recent work has focused on sentence extraction using metadata @xcite , sparse-coding frameworks @xcite , or answer-aware sequential extraction @xcite . our focus is on an answer summarization pipeline which ultimately results in abstractive answer summaries. abstractive answer summarization: another line of work has attempted abstractive answer summarization by treating the tagged best answer as the gold summary of all the other answers (chowdhury and chakraborty, 2019; @xcite . recent work summarizes answers to medical questions via a medical concept graph zhang et al. (2020) and incorporates multi-hop reasoning @xcite and answer relevance from a qa model into the summarization model @xcite . most related to our dataset creation, chowdhury and chakraborty (2019) present cqasumm, a dataset of about 100k automatically-created examples consisting of the best answer as the gold summary, which, however, contains noise due to automatic creation. multi-document summarization: answer summarization can be viewed as a query-based multidocument summarization (mds) problem. approaches to query-focused multi-document summa-rization have dealt with data sparsity via data augmentation @xcite by restructuring the title and paragraphs of news articles to match the target task, coarse to fine-grained modeling @xcite , and by converting generic summarization data into proxy queries @xcite several large-scale mds datasets have been introduced in the news domain @xcite @xcite , for creating wikipedia lead-paragraphs @xcite and for long-form question answering @xcite . however, wikipedia summarization is topic-based and less granular than our setting, and the eli5 dataset @xcite summarizes web documents rather than direct query answers. ## answersumm we introduce our annotation protocol and the characteristics of our manually-curated answer summarization dataset. our annotation pipeline is illustrated in figure 1 . annotation protocol our annotation pipeline consists of four steps 1) answer sentence selection (sentselect), 2) clustering (sentcluster), 3) cluster summarization (clustersumm), and 4) cluster summary fusion (clustersummfusion). we refer to the task of taking forum answers and producing final overall summaries e2esumm. we believe that this pipeline mirrors the process by which humans create summaries of multiple answers by narrowing and organizing information, followed by paraphrasing. furthermore, dividing the summarization task in such a way paves the way for future work in understanding the steps by which a model creates a final summary, and recent work has similarly divided multi-document summarization into these subtasks @xcite . for consistency, the same annotator completes all four steps for a given example. however, we surmise that if each subtask is performed well, then multiple annotators can be involved for a given example. for a given question thread, we present the annotator with the question, the forum from which the question came, the title of the post, and the tags that the original poster associated with the question. the user answers are then presented, where each answer has been automatically segmented into individual sentences using spacy @xcite . it is worth noting that sentence-level granularity is chosen as a simplifying assumption as an appropriate level of segmentation. we are cog-nizant that clause level might be more accurate, however, given state-of-the-art clause detection as well as the precedence for sentence-level modeling in previous work @xcite , we opted for sentence-level segmentation. ## pipeline for data augmentation manually annotating data at the scale of other existing summarization datasets such as cnn-dailymail is impractical. taking advantage of the abundance of unlabeled stackexchange fora available, we develop a pipeline to automatically create data similar to that which is manually annotated above. this process provides augmented data for training summarization models. data filtering similar to filtering for manual annotation, we obtained question threads from stackexchange and applied heuristics motivated by @xcite to find threads suitable for summarization. threads are removed if: 1) there are less than three answers; 2) the longest answer is at least 400 words; 3) the input token length of all answers is not between 100 and 1000 words; and, 4) the average length of answers is between 50 and 300 words. heuristics were chosen to provide enough examples for data augmentation, leaving about 130k question threads in total. pipeline overview the input to our pipeline is a user question and its answers. we select question threads from stackexchange and operate on the sentence-level of these answers, as in our manuallycreated data. our automatic dataset pipeline consists of the following components which aim to mirror the manual pipeline: 1) a relevance model to select relevant sentences and remove irrelevant ones; 2) a clustering model to cluster similar content -reflecting various perspectives; and, 3) input and abstractive summary creation from cluster centroids, resulting in bullet points for the various perspectives reflected in the answers. figure 2 illustrates the pipeline. relevance model: a sentence-level relevance model trained on cqa fora is leveraged to eliminate irrelevant sentences from the input (collection of answers to a question). the output from this stage serves as input to the clustering stage. model details are found in section 6. clustering: typical k-means clustering for short text @xcite @xcite does not work for our setting as the value of k is not known a priori. in fact, it varies from question to question. accordingly, we use the sentence-transformers library @xcite to perform clustering. specifically, we start with a roberta-based model finetuned for sentence embeddings on an entailment dataset, which is further fine-tuned for semantic similarity. clustering parameters are chosen based on a stackoverflow clustering dataset containing labeled clusters, as provided in @xcite . we apply agglomerative clustering with average given a question and answers, relevant sentences are selected and clustered. then, the cluster centroid sentence of non-singleton clusters is removed from the input to use as bullet point summaries. to create the final summaries, we locate the centroid of clusters with at least two sentences and select these centroids as bullet-point summaries. further, we remove the centroid sentences from the sentence-segmented input answers to create a challenging abstractive summarization dataset analogous to the xsum dataset @xcite . since each cluster contains at least two sentences, we assume that given a perfect clustering algorithm, a related sentence can help generate the removed centroid sentence. while removing sentences naturally decreases coherence, we believe that this introduces a tolerable level of noise. we also experimented with cluster centroid paraphrasing and not removing from the input, but this did not improve downstream performance, which we use to measure the value of this dataset and the level of noise. ## rl-based training cross-entropy loss in standard sequence-tosequence model training suffers from exposure bias and also does not directly optimize evaluation metrics @xcite . the reinforce algorithm @xcite , on the other hand, allows for optimizing the evaluation metrics using nondifferentiable rewards. we use an rl multi-reward objective to promote summaries with both high coverage of the input answers and faithfulness. ## results & discussion we provide strong baseline results for the sentselect, clustersumm, clustersummfusion, and e2esumm subtasks of answersumm as a basis for future work. the best results for sentselect are yielded by roberta relevance classification as illustrated in ## conclusion and future work we develop an annotation pipeline for multiperspective answer summarization, introducing the largest human-annotated dataset for this task. we benchmark state-of-the-art models on the content selection, cluster summarization, and end-to-end summarization subtasks of this dataset. we also introduce a pipeline for answer summarization data augmentation that boosts summarization performance. through an analysis of the effects of reinforcement-learning rewards and qualitative examination of model outputs, we point to difficulties in these tasks and areas for future improvement in content selection, abstraction levels, and metrics for model comparison.