Template-based Abstractive Microblog Opinion Summarization (2022)

Abstract: We introduce the task of microblog opinion summarization (MOS) and share a dataset of 3100 gold-standard opinion summaries to facilitate research in this domain. The dataset contains summaries of tweets spanning a 2-year period and covers more topics than any other public Twitter summarization dataset. Summaries are abstractive in nature and have been created by journalists skilled in summarizing news articles following a template separating factual information (main story) from author opinions. Our method differs from previous work on generating gold-standard summaries from social media, which usually involves selecting representative posts and thus favors extractive summarization models. To showcase the dataset’s utility and challenges, we benchmark a range of abstractive and extractive state-of-the-art summarization models and achieve good performance, with the former outperforming the latter. We also show that fine-tuning is necessary to improve performance and investigate the benefits of using different sample sizes.

## introduction social media has gained prominence as a means for the public to exchange opinions on a broad range of topics. furthermore, its social and temporal properties make it a rich resource for policy makers and organizations to track public opinion on a diverse range of issues @xcite @xcite . however, understanding opinions about different issues and entities discussed in large volumes of posts in platforms such as twitter is a difficult task. existing work on twitter employs extractive summarization @xcite @xcite to filter through information by ranking and selecting tweets according to various criteria. however, this approach unavoidably ends up including incomplete or redundant information @xcite . to tackle this challenge we introduce microblog opinion summarization (mos), which we define as a multi-document summarization task aimed at capturing diverse reactions and stances (opinions) of social media users on a topic. while here we apply our methods to twitter data readily available to us, we note that this summarization strategy is also useful for other microblogging platforms. an example of a tweet cluster and its opinion summary is shown in the availability of high quality news article datasets has meant that recent advances in text summarization have focused mostly on this type of data @xcite @xcite . contrary to news article summarization, our task focuses on summarizing an event as well as ensuing public opinions on social media. review opinion summarization @xcite is related to mos and faces the same challenge of filtering through large volumes of user-generated content. while ## defining model baselines as we introduce a novel summarization task (mos), the baselines featured in our experiments are selected from domains tangential to microblog opinion summarization, such as news articles, twitter posts, and product reviews (see section 2). in addition, the selected models represent diverse summarization strategies: abstractive or extractive, supervised or unsupervised, multi-document (mds) or single-document summarization (sds). note that most sds models enforce a length limit (1024 characters) over the input, which makes it impossible to summarize the whole cluster of tweets. we address this issue by only considering the most relevant tweets ordered by topic relevance. the latter is computed using the kullback-leibler divergence with respect to the topical word distribution of the cluster in the gsdmm-lda clustering algorithm @xcite . the summaries were generated such that their length matches the average length of the gold standard. some model parameters (such as lexrank) only allow sentence-level truncation, in which case the length matches the average number of sentences in the gold standard. for models that allow a word limit to the text to be generated (bart, pegasus, t5), a minimum and maximum number of tokens was imposed such that the generated summary would be within [90%, 110%] of the gold standard length. ## evaluation methodology similar to other summarization work @xcite , we perform both automatic and human evaluation of models. automatic evaluation is conducted on a set of 200 clusters: each partition of the test (covid-19 opinionated, covid-19 non-opinionated, election opinionated, election non-opinionated) contains 50 clusters uniformly sampled from the total corpus. for the human evaluation, only the 100 opinionated clusters are evaluated. ## error analysis error analysis is carried out on 30 fine-tuned bart summaries from a set of 15 randomly sampled clusters. the results are found in hallucination fine-tuning on the mos corpus introduces hallucinated content in 8 out of 30 manually evaluated summaries. generated summaries contain opinions that prove to be either false or unfounded after careful inspection of the cluster of tweets. we follow the work of @xcite in classifying hallucinations as either intrinsic (incorrect synthesis of information in the source) or extrinsic (external information error type freq. example intrinsic 4/30 example 1 hallucination generated summary: united states surpasses six million coronavirus cases and deaths and remains at the top of the global list of countries with the most cases and deaths the majority are pleased to see the us still leads the world in terms of cases and deaths, with 180,000 people succumbing to covid-19. extrinsic 4/30 example 2 hallucination generated summary: sex offender rolf harris is involved in a prison brawl after absconding from open jail. the majority think rolf harris deserves to be spat at and called a ''nonce'' and a ''terrorist'' for absconding from open prison. a minority are putting pressure on information 12/30 example 3 loss human summary: miley cyrus invited a homeless man on stage to accept her award. most people thought it was a lovely thing to do and it was emotional. a minority think that it was a publicity stunt. generated summary: miley cyrus had homeless man accept video of the year award at the mtv video music awards. the majority think it was fair play for miley cyrus to allow the homeless man to accept the award on her behalf. she was emotional and selfless. the boy band singer cried and thanked him for accepting the information loss information loss is the most frequent error type. as outlined in @xcite , the majority of current summarization models face length limitations (usually 1024 characters) which are detrimental for long-input documents and tasks. since our task involves the detection of all opinions within the cluster, this weakness may lead to incomplete and less informative summaries, as illustrated in example 3 from ## conclusions and future work we have introduced the task of twitter opinion summarization and constructed the first abstractive corpus for this domain, based on templatebased human summaries. our experiments show that existing extractive models fall short on linguistic quality and informativeness while abstractive models perform better but fail to identify all relevant opinions required by the task. finetuning on our corpus boosts performance as the models learn the summary structure. in the future, we plan to take advantage of the template-based structure of our summaries to refine fine-tuning strategies. one possibility is to exploit style-specific vocabulary during the generation step of model fine-tuning to improve on capturing opinions and other aspects of interest.