Summarizing Community-based Question-Answer Pairs (2022)

Abstract: Community-based Question Answering (CQA), which allows users to acquire their desired information, has increasingly become an essential component of online services in various domains such as E-commerce, travel, and dining. However, an overwhelming number of CQA pairs makes it difficult for users without particular intent to find useful information spread over CQA pairs. To help users quickly digest the key information, we propose the novel CQA summarization task that aims to create a concise summary from CQA pairs. To this end, we first design a multi-stage data annotation process and create a benchmark dataset, COQASUM, based on the Amazon QA corpus. We then compare a collection of extractive and abstractive summarization methods and establish a strong baseline approach DedupLED for the CQA summarization task. Our experiment further confirms two key challenges, sentence-type transfer and deduplication removal, towards the CQA summarization task. Our data and code are publicly available.

## introduction community-based question answering (cqa) enables users to post their questions and obtain answers from other users. with the increase in online services, cqa has become essential for various purposes, including online shopping, hotel/restaurant booking, and job searching. many online platforms implement cqa features to help users acquire additional information about entities (e.g., products, hotels, restaurants, and companies) of their interests. cqa complements customer reviewsanother type of user-generated content, which provide additional information about the entity but mostly focusing on user experiences and their opinions. figure 1 : example of the cqa summarization task. the input contains a collection of qa pairs. duplicated information can be found in a single qa pair or across multiple qa pairs. the output is a concise and coherent summary written in declarative sentences. while cqa greatly benefits users in decisionmaking, digesting information from original question and answer pairs (qa pairs foot_0 ) also has become increasingly harder. due to the community-based nature, cqa tends to have a large number of heavily repetitive qa pairs, which make it difficult for users, especially those who do not have specific intent (i.e., questions), to find and digest key information. existing summarization efforts for cqa @xcite @xcite primarily focus on summarizing answers for a given question, which still assumes that the user has a certain intent. we believe that information spread over qa pairs can be summarized into a more concise text, which helps any users grasp the key points of discussions about a target entity. therefore, we take a step beyond the scope of answer summarization and propose a novel task of cqa summarization, which aims to summarize a collection of qa pairs about a single entity into a concise summary in declarative sentences (shown in figure 1 ). the cqa summarization task has the following two unique challenges. first, cqa summarization needs to solve sentence-type transfer as questions in interrogative sentences have to be converted into declarative sentences to make a concise summary. this challenge is not trivial as existing summarization tasks assume that input and output are both written in declarative sentences. second, cqa contains duplicated questions and answers. that is, different users can post similar questions. a question can have multiple answers, many of which contain duplicate information. also, unlike questionanswering forums (e.g., quora), cqa in online services is less incentivized to remove such redundancy. slightly different questions/answers can provide detailed and useful information not mentioned in other questions/answers. having more similar answers supports the information is more reliable.those properties make existing summarization solutions unsuitable for cqa summarization. to enable further study of the cqa summarization task, we create a corpus coqasum by collecting reference summaries on qa pairs from the amazon qa dataset @xcite . reference summary annotation is challenging for cqa summarization, as a single entity (i.e., a product for the dataset) can have so many questions and answers that the annotator cannot write a summary directly from them. furthermore, the sentence-type difference (i.e., interrogative vs. declarative) obstructs summary writing. to make this annotation feasible, we designed a multi-stage annotation framework. sampled seed qa pairs are given to the annotator to convert into declarative sentences, which are then rewritten into gold-standard summaries by expert writers. at the last step, we collected semantically similar qa pairs to to make the annotated corpus more realistic. we conduct a comprehensive experiment that compares a collection of extractive and abstractive summarization solutions and establish a strong baseline approach, dedupled, for the cqa summarization task. specifically, dedupled finetunes the entire led model for summary generation while additional classifier attached to the encoder is optimized to extract representative qa pairs. leveraging the strengths of both abstractive and extractive summarization objectives, as well as the pre-trained language model checkpoints, dedu-pled significantly outperforms the other alternative methods. our experiment also confirms that dedupled is suitable for cqa summarization, as the model implements the functionality for both (1) sentence-type transfer and (2) duplicate removal. our contributions of the paper are as follows: ## problem definition let d denote a dataset of questions and answers on individual entities {e 1 , e 2 , ..., e |d| } (e.g., products or hotels). for every entity e, we define a set of question-answer pairs qa @xmath0, a i )} |qae| @xmath1, where the question q i and the answer a i are sequences of tokens q @xmath2, ..., w n ) and a @xmath3, ..., a m ) 3 . given a set of qa pairs for an entity e, the cqa summarization task is to generate a natural language summary s e from qa e . ## the coqasum corpus we first describe the multi-stage annotation framework to collect gold-standard reference summaries from input qa pairs and then describe our benchmark dataset coqasum. ## models to examine the feasibility and explore the challenges of cqa summarization, we tested several summarization solutions on coqasum. the models are grouped into extractive, extractive-abstractive and abstractive methods. ## evaluation we conduct comparative experiments to evaluate those models for the cqa summarization task on the coqasum dataset. we randomly split the data into train/validation/test sets, which consist of 1152/144/144 entities, respectively. for lexrank, we limit the output length based on the average reference summary length in the training set. for led and its variations, we fine-tuned the allenai/led-base-16384 checkpoint using the hugging face transformers library. 9 we report the performance of the best epoch (based on rouge-1 f1) chosen on the validation set for all the supervised models. ## analysis 6.1 choice of pre-trained language models to justify our observation that pre-trained language models have strong abilities we test and compare three additional pre-trained language models on coqasum: pegasus @xcite , t5 @xcite , and bart @xcite . we confirm that all models perform better than the extractive and extractive-abstractive models. while pegasus and t5 show similar (24.81 and 24.61 rl, respectively), they are less effective than bart and led (26.89 and 26.01 rl, respectively). ## related work opinion summarization @xcite aims to create a summary from multiple customer reviews. while opinion summarization is relevant to cqa summarization as it summarizes consumergenerated text, customer reviews are significantly different from qa pairs in cqa as they are selfcontained and tend to contain more subjective information. recent opinion summarization models have adopted pre-trained lms (led) for summarizing multiple reviews @xcite . ## conclusion we propose the cqa summarization task to summarize qa pairs in community-based question answering. we develope a multi-stage annotation framework and created a benchmark coqasum for the cqa summarization task. our multi-stage annotation framework decomposes a complex annotation task into three much simpler ones, thus allows higher annotation quality. we further compare a collection of extractive and abstractive summarization methods and establish a strong baseline method dedupled for the cqa summarization task. our experiment also confirms two key challenges, sentence-type transfer and duplication removal, towards the cqa summarization task.