Comprehensive Abstractive Comment Summarization with Dynamic Clustering and Chain of Thought (2024)

Abstract: Real-world news comments pose a significant challenge due to their noisy and ambiguous nature, which complicates their modeling for clustering and summarization tasks. Most previous research has predominantly focused on extractive summarization methods within specific constraints. This paper concentrates on Clustering and Abstractive Summarization of online news Comments (CASC). First, we introduce an enhanced fast clustering algorithm that maintains a dynamic similarity threshold to ensure the high density of each comment cluster being built. Moreover, we pioneer the exploration of tuning Large Language Models (LLMs) through a chain-of-thought strategy to generate summaries for each comment cluster. On the other hand, a notable challenge in CASC research is the scarcity of evaluation data. To address this problem, we design an annotation scheme and contribute a manual test suite tailored for CASC. Experimental results on the test suite demonstrate the effectiveness of our improvements to the baseline methods. In addition, the quantitative and qualitative analyses illustrate the adaptability of our approach to real-world news comment scenarios.

## introduction with the continued development of handheld devices and the boom of social media, the proliferation of online news is swift and relentless. this has fueled heightened interest among researchers on the topic of public opinion analysis, including tasks like opinion mining @xcite @xcite @xcite , sensitive comments detection @xcite @xcite , and news comments summarization @xcite @xcite . this study concentrates on the clustering and abstractive summarization of online news comments (casc). while the past decade has witnessed substantial progress in text summarization across diverse frameworks @xcite @xcite @xcite @xcite @xcite , research specifically targeting casc remains scarce. comment summarization involves extracting a collection of comments either by selecting sentences directly from the comments, referred to as extractive comment summarization, or by compressing the original comments through generative techniques for abstractive summarization. while previous studies have explored a methodology of initially clustering comments into distinct groups followed by summarization @xcite @xcite @xcite , these endeavors have focused on extractive summarization. our investigation of reddit comments reveals a spectrum of characteristics influenced by context, platform, and user demographics. in particular, these comments exhibit diversity in style, length, level of detail, and relevance to the contextual discussion, posing significant challenges for summarization using extractive approaches. in this status quo, this work focuses on abstractive comment summarization with the goal of providing well-organized and extensive coverage of user comments across diverse topics. we confront two main challenges: firstly, the casc task, in its nascent phase, suffers from a lack of a formal task definition, an open-source evaluation scheme, and accessible data. this absence largely hinders the expeditious and streamlined evaluation of diverse methods, models, or algorithms. secondly, although recent large language models (llms) have achieved impressive performance in text sum-marization @xcite @xcite @xcite , there are still limitations when applying such llms to casc. on the one hand, online comments on trending topics can be lengthy, surpassing the strict input sequence length constraints of existing foundational models. on the other hand, while existing llms excel as text generators, their efficacy in clustering remains underdeveloped. in this paper, we propose a novel task on abstractive news comment summarization to address the aforementioned challenges. our contributions encompass three points: quantitative experiments and qualitative analysis of the proposed casc system demonstrate the effectiveness of our improvements over the baseline systems. moreover, a real-scenario case study further indicates the practicability of our approach. previous studies of @xcite are the first to explore extractive summarization of comment clusters, which first group comments into clusters under different topics and then extract comments from each cluster to form the summary. @xcite alternatively proposed a hierarchical clustering method to summarize youtube comments based on comment term normalization and key term extraction. subsequently, @xcite conducted a comprehensive comparison of various clustering methods, including topic-driven clustering, keywordbased clustering, k-means, and cosine distance, demonstrating the superior performance of topicdriven clustering in producing comment clusters suitable for extractive summarization. however, the above topic-oriented approaches often assume that the resulting summary, derived from topically clustered and extracted comments, aligns with user preferences and presents the crux of the discussed matters. such assumptions may not always hold. recognizing this problem, @xcite conducted a series of data analyses and argued that a good comment summary should not only convey the central issues discussed but also encapsulate the opinions surrounding those issues, suggesting that a good summary should cover finer-grained information beyond just topics. in order to capture more informative comment representations, @xcite proposed to utilize neural sentence embedding methods for extractive summarization of comment clusters. despite these efforts have been made in comment cluster summarization, all previous works concentrated on extractive summarization, while the abstractive casc task is largely unexplored. in this paper, we aim to conduct an in-depth study of casc to fill the gap in this field. ## casc test suite this section introduces the proposed annotation scheme coupled with the manually built test suite for casc evaluation. diverging from conventional comment clustering works where the comments are clustered based on the coarse-grained topics, this study focuses more on the finer aspect terms within each news comment. in this paper, we introduce a concept of comment aspect term (cat), which refers to primary objects of the entire discourse, either extracted or abstracted to represent the central focus. the aspect terms serve as the focal points of explicit or implicit opinion expression within the comment. for example, in the following comment, the current russian government doesn't represent us, and we definitely don't want a war with our closest people., the cats are determined as the current russian government and the war, and the comment is expressing disapproval towards the cats. with the above background, we formulate the casc annotation process in three stages: comment aspect term annotation, comment cluster construction, and comment cluster summarization, as detailed below. cat annotation. the process of annotating cat presents two challenges: (1) since aspect terms can be either extracted or abstracted from the original comment, it is difficult to control the granularity of each annotated cat; (2) user comments usually suffer from unclear reference caused by ellipsis or co-reference, which is hard to avoid when reading each comment independently. the two issues may essentially reduce the annotation consistency. in this work, we allow the case that each comment has multiple cats, but we impose a quan-titative limit of fewer than 10 words to control granularity. moreover, to better cover the meaning of each comment, the annotators are instructed to read comments chronologically and refer to the pre-context when labeling aspect terms. the scope of permissible context for reference is limited to a single comment tree (the comment-reply structure). to ensure the quality of annotations and discern optimal candidates, we establish the following rules: (1) preference is given to cats directly extracted from the original comment rather than artificially induced ones; (2) when cats convey the same meaning, we prefer using the same textual expression; (3) aspect terms are not labeled for comments without opinion expression or those devoid of practical meaning. as aspect term annotation inherently involves subjective judgment from annotators, we estimate the consistency of cat annotation by comparing the annotations of 200 news comments between two annotators. we approximate consistency by calculating the word-level coverage between each pair of summaries, resulting in a score of 43.67%. 1 comment cluster annotation. manually clustering a large volume of comments from scratch, depending on the limited memory of human annotators, is time-consuming and challenging, fraught with difficulties in ensuring annotation quality. as stated before, we study clustering comments based on aspect terms. specifically, we consider semantic affinity inferred from the frequency of word co-occurrence between cats as a reliable indicator of similarity. first, we represent each comment by averaging the glove vectors within the manually annotated aspect terms. then, the comments are clustered based on the cosine distance between aspect term representations through fast clustering with a high similarity threshold of 0.85. comments lacking manual aspect terms are assigned to a single unique cluster, labeled as trivial. lastly, annotators manually check the correctness of these clusters according to the following guidelines: (1) given a cluster @xmath0, . . . , c n } with n comments, if the aspect terms and content of c k significantly differ from others in the cluster, exclude c k from g and assign it to a suitable comment cluster or the trivial group. (2) if cluster g i shares a closely related topic with cluster g j , combine them into a unified cluster. (3) check comments annotated with aspect terms in the trivial group to determine if they can be reallocated to an existing non-trivial cluster. comment summary annotation. text summarization annotation has long been contentious, primarily due to the inherent subjectivity of annotators, which can easily influence the standard of summary annotation. recent studies @xcite @xcite have demonstrated that llm summarizers can perform on par with or better than human summarizers. among them, the research in @xcite shows that when quantitatively and qualitatively assessed against human annotators, llm summaries with increased factuality are notably favored by human evaluators. inspired by this, we transition from manual summary annotation to llm-assisted summary annotation. specifically, based on the manually adjusted comment clusters, we first use gpt3.5 to generate an llm summary for each comment cluster. then, we manually postedit the summaries and refine those suffering from hallucinations to produce the final test suite. annotated data overview. based on comments of news articles sourced from reddit and new york times 2017, foot_1 our annotated dataset contains 11 article clusters (acs) covering various topics, which involve a total of 20 news articles and 7,958 comments. guided by our annotation guidelines and with the assistance of four annotators, we constructed 848 comment clusters (ccs) along with their corresponding summaries. further details are shown in ## analysis & discussion comment cluster distribution. figure 1 presents the distribution of 1,467 comment clusters constructed by our casc system. the column bars labeled with tag ① depict the distribution of comment cluster sizes, the poly-line marked with tag ② denotes the similarity threshold applied to each cluster, and the poly-line labeled with ③ indicates the density value of each cluster. the diagram shows that as the cluster size increases, our dynamic algorithm progressively elevates the similarity threshold, to impose stringent criteria for cluster formation. since a comment is included in the cluster only when its similarity to the centroid comment surpasses the threshold, the cluster density is distributed above the threshold line, with higher similarity thresholds leading to increased cmt-1. how do you think you came in contact with flu? cmt-2. so, if i caught it, do you think i would know or would i just think it was the regular flu? cmt-3. i am currently trying to avoid other humans. although, that sounds like a good revenge for some people. i probably picked up the flu in a public place. cmt-4. how would you explain the news describing people with "mild symptoms". do they mean mild compared to regular flu, or mild compared with dying? cmt-5. or maybe from a patient who was there for something else and was unknowingly serving as an incubater for the disease. excuse me if you've already answered this, but were you not given every possible vaccination, considering your profession? and even if none are effective, are health-care workers continuing to take every precaution against infection of this flu, or have things been a bit laxer lately? do you mind giving the region of texas in which you work? auto aspect terms: flu, contact, symptoms, mild symptoms, vaccination, health-care workers. summary w/o bg: the user comments revolve around various aspects related to the covid-19 pandemic. users express concerns about how they may have contracted the flu and how they would know if they had it if they caught it. they also discuss symptoms and severity levels of the virus as well as health-care worker precautions taken against it. some users speculate about potential incubators of the disease such as patients who may have served as incubators for others. overall, there is a focus on understanding how best to prevent the flu. summary w/ bg (h1n1 virus, swine flu): the user comments revolve around various aspects related to the h1n1 virus and swine flu. users express concerns about how they may have contracted the flu and how they would know if they had it compared to regular flu symptoms. they also discuss how mild symptoms are interpreted and whether they refer to a comparison with dying or if they indicate an unknowingly serving as an incubater for the disease. additionally, users question whether health-care workers are taking proper precautions against infection and if things have become laxer. ## conclusion this paper presents an initial exploration into clustering and abstractive summarization of online news comments (casc). we introduce a manual annotation scheme, create a test suite, and propose an evaluation metric tailored for the casc task. on this basis, we introduce a dynamic clustering algorithm and a novel ranking mechanism for effective comment clustering. in particular, we perform the first practice of tuning foundation models for comment cluster summarization. to foster further studies, the data and codes will be released to the research community upon application.