Unsupervised Approach to Multilingual User Comments Summarization (2021)

Abstract: User commenting is a valuable feature of many news outlets, enabling them a contact with readers and enabling readers to express their opinion, provide different viewpoints, and even complementary information. Yet, large volumes of user comments are hard to filter, let alone read and extract relevant information. The research on the summarization of user comments is still in its infancy, and human-created summarization datasets are scarce, especially for less-resourced languages. To address this issue, we propose an unsupervised approach to user comments summarization, which uses a modern multilingual representation of sentences together with standard extractive summarization techniques. Our comparison of different sentence representation approaches coupled with different summarization approaches shows that the most successful combinations are the same in news and comment summarization. The empirical results and presented visualisation show usefulness of the proposed methodology for several languages.

## introduction readers of news articles are often interested in what others think, what their perspectives are, and whether they can get any additional information from them. user comment sections on news web pages are often a good source for extending, presenting, and challenging their own views. on the other hand, many news providers see user comments sections of their websites as a way to connect to their readers, get relevant feedback, and sometimes even extract complementary information. many news articles get a large number of comments in a short time, which is especially true for popular and controversial topics. when dealing with an individual article, users can usually sort comments by relevancy or publishing time. while not ideal, this is satisfactory to get insight into the most popular thread or discussion but lacks in providing an overview of the whole discussion @xcite . this, together with the low amount of time users are willing to spend in reading comments, is one of the reasons to automatically provide comprehensive overviews of discussions. user comments can be irrelevant, deceiving, and may contain hate speech. language is often informal with ill-formed sentences full of spelling and grammatical errors that are hard to understand. because of that, comments are easily dismissed as not worth the attention and time. in addition, non-standard expressed content is difficult to encode into an informative numerical representation as standard embedding techniques are mostly based on more standard language @xcite . the goal of text summarization is to compress original data and present it in a shorter form conveying the essential information @xcite . two main approaches exist, extractive and abstractive. the extractive summarization approach selects essential information and does not modify content; its goal is to copy the most informative non-redundant sentences, phrases, or other units of a text. the abstractive approach is similar to how humans summarise documents. it may use new words and expressions, compress long sentences, combine multiple sentences, replace phrases, etc. current neural network based abstractive approaches mostly provide useful and fluent summaries for short texts but offer no guarantee concerning text correctness @xcite . news article summarization is a well-defined and the most studied task within the field of automatic text summarization with several available datasets suitable for supervised learning @xcite . for this task also several unsupervised methods exist, based on graph centrality approaches or clustering. on the other hand, the user comment summarization task is not well-defined and established. in a survey paper on user comments, @xcite describe it as the extraction of sentences that express an opinion. this proposal categorises it as an information retrieval task, close to comment filtering and comment ranking. we believe that this categorisation is limited as it does not consider many other aspects, such as complementarity of information, coverage of different topics and opinions, impact on public discourse, possibly offensive speech, non-standard language, etc. cross-lingual approaches to text processing @xcite enable the transfer of trained models from resource-rich languages to lowresource languages. many multilingual neural sentence representation models were released @xcite @xcite , which presents an opportunity to improve standard unsupervised extractive approaches @xcite @xcite that use sparse representations such as tf-idf weighted bag-of-words. in this work, we developed an unsupervised extractive approach to text summarization that combines traditional unsupervised methods (graph and clustering-based) with the above-mentioned stateof-the-art multilingual sentence encoders. we assess these encoders in combination with different extractive summarizers and dimensionality reduction techniques. we used croatian, english and german datasets containing news articles and user comments. our main contributions are: the paper consists of six sections. in section 2, we present the related work. section 3 contains description of datasets we used. in section 4, we outline and explain our approach to unsupervised text summarization. section 5 presents visual and automatic evaluation of the results. in section 6, we summarize the work done, present limitations of our approach, and ideas for further work. ## related work in this section, we present related research on comment summarization and other related summarization tasks. user comments can be divided into comments on non-textual resources (photos or videos) and comments on textual resources (news articles, product reviews, etc.) @xcite . @xcite argue that the most important tasks done on comments are filtering, ranking, and summarization. we focus on the latter two. most of the research on user comments summarization uses unsupervised extractive approaches that combine ranking and clustering methods. @xcite used lda for clustering, and ranking algorithms (mead, lexrank) to summarize comments on youtube videos. ma et al. (2012) developed a topic-driven approach in which they compared clustering methods and ranking methods (maximal marginal relevance, rating & length) on comments from yahoo news. llewellyn et al. (2014) used standard clustering and ranking methods (k-means, pagerank, etc.) to summarize the comments section of the uk newspaper the guardian. hsu et al. (2011) proposed a hierarchical comments-based clustering approach to summarize youtube user comments. all listed methods use classical text representation approaches, while we propose the use of modern neural sentence embedding methods. a related task to comment summarization is discussion thread summarization. the distinctive difference is that original posts are very different from news articles. van oortmerssen et al. ( 2017 ) used text mining to analyze cancer forum discussions. in addition to ranking and clustering, @xcite use hand-crafted text quality features such as common words between the thread reply and the initial post, a semantic distance between thread reply and thread centroid, etc. the conversation summarization @xcite , email summarization @xcite , and twitter topics summarization @xcite are also relevant related tasks. ## datasets in this section, we first describe the creation of two croatian summarization datasets used in our research: news articles, and user comments. we also present english and german dataset of user comments. the cronews summarization dataset was created from the corpus of approximately 1.8 million news articles from the popular croatian 24sata news portal foot_0 . the second dataset (crocomments) is a small evaluation dataset @xcite and contains user comments of 42 articles from croatian večernji list website foot_1 , together with their short human-written abstractive summaries foot_2 . we preprocessed the news articles from the news corpus into a one-sentence-per-line form using the croatian tokenizer available in the stanza nlp package @xcite . the user comments in crocomments were already preprocessed in a similar way @xcite . the articles in the original news dataset contained no summaries. we took the first paragraph of an article as a proxy for a summary. in the dataset, this paragraph is named 'lead'. we sampled 5000 (from a total of 17 194) examples that satisfied the next criteria: more than 6 and less than 30 sentences were present in an article (we presupposed that articles with less than 6 sentences are too short for summarization), and the overlap between the abstract (lead) and article text was within 40 and 90 rouge-l points. the last criterion was designed to make sure that the first paragraph of an article overlaps with the rest of it in terms of content but we avoided strictly duplicated content. most of the abstracts have a missing period at the end. we fixed that by appending it at the end of an article. we call the resulting dataset cronews in the remainder of the paper. while we focused on the croatian language, to assess the multilingual potential of the proposed approach, we tested it also on english and german. for english, we used the new york times comments corpus foot_3 with over 2 million comments. for german, we used one million posts corpus (schabus and skowron, 2018) with 1 million comments from the austrian daily broadsheet newspaper der standard. ## methodology in this section, we describe our approach to unsupervised (multilingual) summarization which is comprised of two main components: clustering approaches group similar sentence vectors and select the representative sentences based on the proximity to the centroid vector. graph-based methods construct a graph based on the similarity of sentence vectors and then use graph node rankings to rank the sentences. the best-ranked sentences are returned as the summary. as a further, optional component of our approach, the sentence vectors can be mapped to twodimensional space with dimensionality reduction techniques (we use pca or umap) and visualized in an interactive graph. to demonstrate these capabilities, we released a jupyter notebook on google colab foot_4 . ## evaluation in this section, we first provide visualization of sentence embeddings, followed by the analysis of summarization. the visualization demonstrates the suitability of the proposed cross-lingual sentence representation for unsupervised summarization. in summarization experiments, we first present results of news article summarization, followed by the commentaries. ## conclusion we developed a multilingual unsupervised approach to user commentary summarization and tested it on a less-resourced croatian language. our models are based on cross-lingual neural sentence encoders, which make them easily applicable to many languages with little or no preprocessing. we tested several sentence representations and assessed the effect of dimensionality reduction. we used clustering and graph-based ranking algorithms to select sentences that form the final summaries. the results were promising both on the news articles dataset and the user comments evaluation dataset. the source code of our approach is freely available under the open-source licence. the presented approach has several limitations. it only works within extractive summarization approaches, which do not allow sentence modification. with abstraction techniques, e.g., sentence compression, we could further distill the important information. we only tested sentence representation methods, while paragraph or document embeddings would also be sensible. we also did not exploit the information contained in the threading structure of the commentaries and possible relation of comments with the text of an original article. in further work, we intend to exploit additional information in comments which was not used in the present study. the number of likes that a comment received could be used to weight sentences. instead of working on a sentence-level, we could take a comment as a whole and embed it as a document. we plan to extend the work on visualization since it showed promising results, especially in the interactive exploration mode, inaccessible in the paper format. id text 24107006 score: 0.0282 this art is all about perception . it is about the point the artist is trying to make and how the viewer sees it . this art should not be limited because it is attached to an emotion these moments being recorded through art of a society that claims to be post racial opens the eyes of those who do not want to see and forces them to . this illustration does that and in my opinion that makes it so much more valuable because it does not just sit in silence it sends a message . 23235619 score: 0.0283 artists should n't be limited or restricted in what they can do as an artist . everyone should have a voice or take on a matter no matter how unpopular or offensive the opinion is . censoring art defeats the creativity and free expression in art . censorship perverts the message the artist try 's to convey . 22099108 score: 0.0273 i believe that all subjects should be fair game for an artist . it should n't matter if they are depicting a murder , or even if it 's " black subject matter " , every artist has a voice that deserves to be heard . as smith writes " we all encounter art we do n't like , that upsets and infuriates us . " ( 1 ) i understand that some topics are difficult to talk about and that some art is can cause anger but i think that it is irrational to make topics off -limits because people do n't agree with it . 22098876 score: 0.0264 i personally believe that artists should be able to write about anything they want , drive to the studio , then turn those words into beautiful music . music is an art and in art there are no limits so honestly whatever they feel is relevant to write about , they should have the freedom to do so . regardless of peoples personal opinions artist should be comfortable to talk about what they want to talk about . " we all encounter art we do n't like , that upsets and infuriates us . " ( gilpin , 1 ) i understand that some subjects are very sensitive , but most of the things people do n't like to hear are usually cold hard facts about the dark side of society . a few examples would be , hate crimes against all races , racism in america , people killing other people . it s just the sad truth that a lot of people hate to hear . music is a powerful -subject that can really impact a person . 22075721 0.0258 nothing should be in limited to artist . they should have the freedom to do what they pleased . 22054073 0.0252 i believe there is n't a problem when a white artist draws a topic that is related to discrimination against the blacks . this artist may want to show that killing black people is wrong . it does n't matter if she 's white or black . 22041906 score: 0.0280 i do n't think that any topic is out of bounds to an artist . that is the idea of an artist , is n't it ? to talk about subjects that they think should be talked about , or that they feel motivated to bring attention to . i do n't think it is right to throw blame and anger towards one group because they are creating art about a different group . i understand why there is anger , but demanding that a work be destroyed is just absurd to me . could the artist have done something differently ? possibly , but demanding empathy and understanding from a group different than your own , and then saying their act of trying to do so is inappropriate just does n't make sense . i do n't think any one group " owns " history . history is a human experience . people as a collective own the histories that shaped the world they live in . that is the point of the exhibition . the exhibition description on the whitney site says , " throughout the exhibition , artists challenge us to consider how these realities affect our senses of self and community . " instead of focusing on the color of the artists skin , we should be focusing on the point of the show .. how the painting makes us feel about ourselves and our communities , because i am sure that everyone could say that there is room for improvement when it comes to both . 22031632 score: 0.0219 the question of whether or not any group " owns " a portion of history is not the issue . it is about how that imagery is used , if it is used intelligently , and that it mimics an aspect of white racism : the historic practice of whites displaying the mutilated corpses of black people . to make the issue about censorship is to miss the point . instead students should be asked to consider how a white person might have better handled her desire to show empathy . günes erkan and dragomir r radev. 2004. lexrank: graph-based lexical centrality as salience in text summarization. journal of artificial intelligence research, 22:457-479.