News Aggregation with Diverse Viewpoint Identification Using Neural Embeddings and Semantic Understanding Models (2020)

Abstract: Today’s news volume makes it impractical for readers to get a diverse and comprehensive view of published articles written from opposing viewpoints. We introduce a transformer-based news aggregation system, composed of topic modeling, semantic clustering, claim extraction, and textual entailment that identifies viewpoints presented in articles within a semantic cluster and classifies them into positive, neutral and negative entailments. Our novel embedded topic model using BERT-based embeddings outperforms baseline topic modeling algorithms by an 11% relative improvement. We compare recent semantic similarity models in the context of news aggregation, evaluate transformer-based models for claim extraction on news data, and demonstrate the use of textual entailment models for diverse viewpoint identification.

## introduction the advent of news aggregators has ushered in a new age of information, exposing readers to continuous streams of articles from diverse outlets. however, the proliferation of data makes finding viewpoints presented in different articles challenging. we introduce a novel transformer-based news aggregation system that identifies diverse viewpoints, depicted in figure 1 . rather than use preset criteria or learned behavior, our system provides a list of viewpoints covered in news articles and allows users to decide which viewpoints to explore. our system consists of the following components, illustrated in figure 2: (i) topic modeling organizes articles from multiple news sources into clusters, (ii) hypothesis extraction extracts an opinionated summary sentence (i.e., hypothesis) from each article, (iii) semantic similarity identifies differing viewpoints (i.e., sub-clusters) within each topic based on the hypotheses, (iv) premise extraction extracts a summary sentence (i.e., premise) from a group of articles associated with a viewpoint, (v) textual entailment evaluates the entailment between the hypothesis of each article and the premise of its subcluster. as part of this work, we define hypothesis extraction and premise extraction as subsets of claim extraction, where a claim is defined as a sentence expressing viewpoints associated with a news article. we define a hypothesis as a single summary sentence that represents an article's viewpoint, and use the terms hypothesis extraction and single-document subjectivity analysis interchangeably. we define a premise as a single summary sentence that represents viewpoints shared by multiple articles, and use the terms premise extraction and multi-document subjectivity analysis interchangeably. their system expects a claim as input to identify relevant documents, select sentences as evidence from the document, and finally classify the claim. other authors evaluate the performance of transformerbased models against baseline models for debate data @xcite @xcite al., 2020 2020 ), which has applications for news data but is structured slightly differently. a single news article can be clustered under multiple topics and report multiple opinions within the same article. in contrast, debate data often directly align with one particular pre-defined topic and involve separate opinions. our system differs from an argument search engine with indexing and retrieval @xcite . to adjust to the dynamic incoming stream and multiple sources of news data, we explore the generalization capability of language models to automate the news aggregation and viewpoint discovery problem. we have chosen to develop our own labeled article data set targeted specifically for news applications. topic modeling: a common approach to clustering documents based on textual content is multinomial latent dirichlet allocation (lda) @xcite . many alternatives have been investigated to improve semantic coherence by representing words with word2vec embeddings @xcite . one such approach is the embedded topic model (etm) (dieng et al., 2020), which uniquely represents each document as latent topics, where each topic is an embedding in the semantic space of the words. in this paper, we use etm for our topic modeling and investigate using transformer-based embeddings in lieu of word2vec embeddings to improve quality of clustering. semantic similarity: semantic textual similarity (sts) refers to the goal of quantifying the degree of similarity between two bodies of text by capturing the degree to which the meanings of the two inputs overlap @xcite . until recently, state-of-the-art sts systems have relied heavily on word embedding approaches @xcite , which lack the capability to fully capture semantic context. methods such as infersent were developed as a solution to embed multiples words, phrases, or sentences into a single representation @xcite . the universal sentence encoder (use) models @xcite , bert @xcite , and other models such as roberta @xcite and gpt-3 @xcite have since made significant improvements on infersent. ## methods a demo of our system is available at https://harvard-almit.github.io/newsaggregator/ . we tested this system on a dataset of over 1,000 scraped articles from various news outlets, such as yahoo news and the post and courier. figure 2 illustrates how news articles are processed across the components in our system: (i) topic modeling generates cluster ids for the articles, (ii) hypothesis extraction generates hypotheses for the clustered articles, (iii) semantic similarity generates subcluster ids using hypotheses, (iv) premise extraction generates premises for articles in each subcluster, and (v) textual entailment, consuming outputs of hypothesis extraction and premise extraction, generates entailment predictions and provides the results of our news aggregation system. we perform topic modeling with three approaches using 20 newsgroups data @xcite and adjusted rand index scoring @xcite : (i) latent dirichlet allocation (lda) using the gensim package @xcite , (ii) etm with word2vec embeddings, and (iii) etm with centroids of bert based embeddings @xcite @xcite , where we select a value for num bert centroids as the number of embeddings etm will use, and use k-means clustering (@xmath0. for training hypothesis extraction models on the clustered articles, we use a modified version of the mpqa opinion corpus v3.0 consisting of expressive subjective elements @xcite . we train a multinomial naïve bayes classifier and fine-tune bert, xlnet @xcite , and albert @xcite models using huggingface's transformers library @xcite on pre-processed mpqa data for sentence-level subjectivity analysis (i.e., binary opinion classification of sentences). the semantic similarity module then clusters generated hypotheses of documents within a topic into clusters of semantically related articles, in which each article is associated with a single, more specific topic. we considered bert, roberta, and distilbert @xcite in a siamese network structure (reimers and gurevych, 2019), in addition to use using the pearson correlation coefficient, for the semantic similarity module. the models were fine-tuned on the argument facet similarity corpus by @xcite and the sts-benchmark dataset provided by the senteval (conneau and kiela, 2018) package. for premise extraction, we fine-tuned a large bart model (406m parameters) @xcite and a small t5 model (60m parameters) @xcite using the transformers library on data taken from ibm's project debater claim stance dataset @xcite . we reformatted this dataset into a summarization dataset for fine-tuning. we chose not to utilize the claims provided in the dataset, since the hypothesis extraction module already accomplishes this purpose, and used topics (statements that represent a group of articles) from the dataset instead. the final output of the system is provided by the textual entailment module. for each article in a semantic similarity sub-cluster, premises and hypotheses generated from the claim extraction module are input to the textual entailment module to predict whether the hypothesis contradicts, entails, or is unrelated to the premise. we evaluated fairseq's pre-trained roberta and bart models fine-tuned for mnli @xcite using the transformers library for textual entailment on the claim stance dataset presented by @xcite , given that bart reportedly performs similar to roberta on the mnli task @xcite . ## discussion we found topics generated by etm for our dataset were more coherent compared to lda for topic modeling. etm with bert based embeddings outperforms etm with word2vec embeddings when etm is trained with a number of bert centroids greater than the number of word2vec embeddings associated with the corpus. training etm with bert centroids involves significantly more computational work than training etm with word2vec embeddings. on the other hand, in settings where both model creation and predictions are based on the same, full dataset, etm with bert-based embeddings does perform better and could be incorporated. we demonstrate that hypothesis extraction can be phrased as subjectivity analysis, and we found xlnet and albert can deliver high performance for this task. a novel aspect of our methodology is that we employ the generated hypotheses as input to the semantic similarity module. current state-of-the-art models treat semantic similarity as a pair-wise regression problem, making them computationally inefficient for clustering for news aggregation. we found that transformer-based models increased the quality of the clustering compared to use. the results here show that we can efficiently find semantic clusters with standard clustering methods, e.g., k-means++ @xcite , or density based clustering, e.g., dbscan @xcite , to present users with a diverse set of articles on a specific topic. we show that premise extraction is closely related to the task of abstractive summarization, as premises must be constructed to enable a group of articles to agree or disagree with statements. we observed the small t5 model was able to significantly outperform the larger bart model. we demonstrate that multidocument and single-document claim extractions can be informative premises and hypotheses that are inputs to a textual entailment module. from the second premise-hypothesis pair in section 4, we see there is a small contradiction that the model does not detect, but could potentially predict if a different premise-hypothesis pair was chosen, or if predictions were validated using phrases from the hypothesis (e.g., the model based its prediction on the phrase "what looks like an official website asking for personal information"). consequentially, we found that different premise-hypothesis pairs within an article can lead to different predictions from the textual entailment module for the same article, due to opposing viewpoints described in the article and distinct word phrasing between sentences. ## conclusion we have introduced a transformer-based news aggregation system, consisting of topic modeling, hypothesis extraction, semantic clustering, premise extraction, and textual entailment that allows readers to view articles from diverse viewpoints. our results show relative improvements over baseline models in the range of 10-23% using embedded topic modeling, semantic similarity through fine-tuned bert models with a siamese network structure, and hypothesis extraction using large pre-trained language models for sentence-level subjectivity analysis. our results also show a five-fold loss decrease when using a small t5 model, compared to a large bart model, for premise extraction. the system we have developed demonstrates that pre-trained bert-based models of textual entailment can be used to identify diverse viewpoints.