Event-Driven News Stream Clustering using Entity-Aware Contextual Embeddings (2021)

Abstract: We propose a method for online news stream clustering that is a variant of the non-parametric streaming K-means algorithm. Our model uses a combination of sparse and dense document representations, aggregates document-cluster similarity along these multiple representations and makes the clustering decision using a neural classifier. The weighted document-cluster similarity model is learned using a novel adaptation of the triplet loss into a linear classification objective. We show that the use of a suitable fine-tuning objective and external knowledge in pre-trained transformer models yields significant improvements in the effectiveness of contextual embeddings for clustering. Our model achieves a new state-of-the-art on a standard stream clustering dataset of English documents.

## introduction human presentation and understanding of news articles is almost never isolated. seminal real-world events spawn a chain of strongly correlated news articles that form a news story over time. given the abundance of online news sources, the consumption of news in the context of the stories they belong to is challenging. unless people are able to scour the many news sources multiple times a day, major events of interest can be missed as they occur. the real-time monitoring of news, segregating articles into their corresponding stories, thus enables people to follow news stories over time. this goal of identifying and tracking topics from a news stream was first introduced in the topic detection and tracking (tdt) task @xcite . topics in the news stream setting usually correspond to real-world events, while news articles may also be categorized thematically into * work done during internship at amazon sports, politics, etc. we focus on the task of clustering news on the basis of event-based story chains. we make a distinction between our definition of an event topic, which follows tdt and refers to large-scale real-world events, and the fine-grained events used in trigger-based event detection @xcite . given the non-parametric nature of our task (the number of events is not known beforehand and evolves over time), the two primary approaches have been topic modeling using hierarchical dirichlet processes (hdps) @xcite and stream clustering @xcite @xcite . while hdps use word distributions within documents to infer topics, stream clustering models use representation strategies to encode and cluster documents. contemporary models have adopted stream clustering using tf-idf weighted bag of words representations to achieve state-of-the-art results @xcite . in this paper, we present a model for event topic detection and tracking from news streams that leverages a combination of dense and sparse document representations. our dense representations are obtained from bert models @xcite finetuned using the triplet network architecture @xcite on the event similarity task, which we describe in section 3. we also use an adaptation of the triplet loss to learn a support vector machine (svm) @xcite based document-cluster similarity model and handle the non-parametric cluster creation using a shallow neural network. we empirically show consistent improvement in clustering performance across many clustering metrics and significantly less cluster fragmentation. the main contributions of this paper are: in this section, we introduce the tdt task, prior work on tracking events from news streams and a few related parametric variants of the tdt task. the goal of the tdt task is to organize a collection of news articles into groups called topics. topics are defined as sets of highly correlated news articles that are related to some seminal real-world event. this is a narrower definition than the general notion of a topic which could include subjects (like new york city) as well. tdt defines an event to be represented by a @xmath0, time, people involved>, which spawns a series of news articles over time. we are interested in all five sub-tasks of tdt -story segmentation, first story detection, cluster detection, tracking and story link detection -though we do not explicitly tackle these sub-problems individually. after the initial work on the tdt corpora, interest in news stream clustering was rekindled by the news tracking system newslens @xcite . newslens tackled the problem in multiple stages: (1) document representation through keyphrase extraction; (2) non-parametric batch clustering using the louvian algorithm @xcite ; and (3) linking of clusters across batches. @xcite presented a modified version of this model, using tf-idf bag of words document representations instead of keywords. they also compared the relative performance of sparse tf-idf bag of words and dense doc2vec @xcite representations and showed that the latter performs worse, both individually and in unison with sparse representations. linger and hajaiej (2020) extended this batch clustering idea to the multilingual setting by incorporating a siamese multilingual-distilbert @xcite model to link clusters across languages. in contrast to the batch-clustering approach, @xcite adopt an online clustering paradigm, where streaming documents are compared against existing clusters to find the best match or to create a new cluster. we adopt this stream clustering approach as it is robust to temporal density variations in the news stream. batch clustering models tune a batch size hyperparameter that is both training corpus dependent and might not be able to adjust to temporal variations in stream density. in their model, they also use a pipeline architecture, having separate models for document-cluster similarity computation and cluster creation. similarity between a document and cluster is computed along multiple document representations and then aggregated using a rank-svm model @xcite . the decision to merge a document with a cluster or create a new cluster is taken by an svm classifier. our model also follows this architecture, but critically adds dense document representations, an svm trained on the adapted triplet loss for aggregating document-cluster similarities and a shallow neural network for cluster creation. news event tracking has also been framed as a non-parametric topic modeling problem @xcite and hdps that share parameters across temporal batches have been used for this task @xcite . dense document representations have been shown to be useful in the parametric variant of our problem, with neural lda @xcite @xcite , temporal topic evolution models @xcite @xcite and embedding space clustering @xcite being some prominent approaches in the literature. ## methodology our clustering model is a variant of the streaming k-means algorithm (macqueen, 1967) with two key differences: (1) we compute the similarity between documents and clusters along a set of representations instead of a single vector representation; and (2) we decide the cluster membership using the output of a neural classifier, a learned model, instead of a static tuned threshold. at any point in time t, let n be the number of clusters the model has created thus far, called the cluster pool. given a continuous stream of news documents, the goal of the model is to decide the cluster membership (if any) for each input document. in our task, we assume that each document belongs to a single event, represented by a cluster. the architecture of the model, as shown in figure 1 , consists of three main components : (1) document representations, (2) document-cluster similarity computation using a weighted similarity model and (3) cluster creation model. in what follows, we describe each of these components. ## conclusion in this paper, we present a novel news stream clustering algorithm that uses a combination of sparse and dense vector representations. we show that while dense embeddings by themselves do not achieve the best clustering results, enhancements like entity awareness and event similarity finetuning make them effective in conjunction with sparse and temporal representations. our model achieves new state-of-the-art results on the @xcite dataset. we also analyze the problem of cluster fragmentation noting that our approach is able to produce a similar number of clusters as in the test set, in contrast to prior work which produces far too many clusters. we note issues with the b-cubed metrics and we complement our results using ceaf-e as an additional metric for our clustering task. in addtion, we provide a comprehensive empirical evaluation across many metrics to show the robustness of our model to metric idiosyncrasies.