Evaluation of Thematic Coherence in Microblogs (2021)

Abstract: Collecting together microblogs representing opinions about the same topics within the same timeframe is useful to a number of different tasks and practitioners. A major question is how to evaluate the quality of such thematic clusters. Here we create a corpus of microblog clusters from three different domains and time windows and define the task of evaluating thematic coherence. We provide annotation guidelines and human annotations of thematic coherence by journalist experts. We subsequently investigate the efficacy of different automated evaluation metrics for the task. We consider a range of metrics including surface level metrics, ones for topic model coherence and text generation metrics (TGMs). While surface level metrics perform well, outperforming topic coherence metrics, they are not as consistent as TGMs. TGMs are more reliable than all other metrics considered for capturing thematic coherence in microblog clusters due to being less sensitive to the effect of time windows.

## introduction as social media gains popularity for news tracking, unfolding stories are accompanied by a vast spectrum of reactions from users of social media platforms. topic modelling and clustering methods have emerged as potential solutions to challenges of filtering and making sense of large volumes of microblog posts @xcite @xcite . providing a way to access easily a wide range of reactions around a topic or event has the potential to help those, such as journalists @xcite , police @xcite , health @xcite and public safety professionals @xcite , who increasingly rely on social media to detect and monitor progress of events, public opinion and spread of misinformation. recent work on grouping together views about tweets expressing opinions about the same entities has obtained clusters of tweets by leveraging two topic models in a hierarchical approach @xcite . the theme of such clusters can either be represented by their top-n highest-probability words or measured by the semantic similarity among the tweets. one of the questions regarding thematic clusters is how well the posts grouped together relate to each other (thematic coherence) and how useful such clusters can be. for example, the clusters can be used to discover topics that have low coverage in traditional news media @xcite . @xcite employ the centroids of twitter clusters as the basis for topic specific temporal summaries. the aim of our work is to identify reliable metrics for measuring thematic coherence in clusters of microblog posts. we define thematic coherence in microblogs as follows: given clusters of posts that represent a subject or event within a broad topic, with enough diversity in the posts to showcase different stances and user opinions related to the subject matter, thematic coherence is the extent to which posts belong together, allowing domain experts to easily extract and summarise stories underpinning the posts. to measure thematic coherence of clusters we require robust domain-independent evaluation metrics that correlate highly with human judgement for coherence. a similar requirement is posed by the need to evaluate coherence in topic models. @xcite provide a framework for an extensive set of coherence measures all restricted to word-level analysis. @xcite show that adding contextual information to neural topic models improves topic coherence. however, the most commonly used word-level evaluation of topic coherence still ignores the local context of each word. ultimately, the metrics need to achieve an opti-mal balance between coherence and diversity, such that resulting topics describe a logical exposition of views and beliefs with a low level of duplication. here we evaluate thematic coherence in microblogs on the basis of topic coherence metrics, while also using research in text generation evaluation to assess semantic similarity and thematic relatedness. we consider a range of state-of-the-art text generation metrics (tgms), such as bertscore @xcite , moverscore @xcite and bleurt @xcite , which we repurpose for evaluating thematic coherence in microblogs and correlate them with assessments of coherence by journalist experts. the main contributions of this paper are: ## related work measures of topic model coherence: the most common approach to evaluating topic model coherence is to identify the latent connection between topic words representing the topic. once a function between two words is established, topic coherence can be defined as the (average) sum of the function values over all word pairs in the set of most probable words. newman et al. (2010) use pointwise mutual information (pmi) as the function of choice, employing co-occurrence statistics derived from external corpora. mimno et al. (2011) subsequently showed that a modified version of pmi correlates better with expert annotators. alsumait et al. (2009) identified junk topics by measuring the distance between topic distribution and corpus-wide distribution of words. fang et al. ( 2016a ) model topic coherence by setting the distance between two topic words to be the cosine similarity of their respective embedded vectors. due to its generalisability potential we follow this latter approach to topic coherence to measure thematic coherence in tweet clusters. we consider glove @xcite and bertweet (nguyen et al., 2020) embeddings, derived from language models pretrained on large external twitter corpora. to improve performance and reduce sensitivity to noise, we followed the work of lau and baldwin (2016), who consider the mean topic coherence over several topic cardinalities |w | ∈ {5, 10, 15, 20}. another approach to topic coherence involves detecting intruder words given a set of topic words, an intruder and a document. if the intruder is identified correctly then the topic is considered coherent. researchers have explored varying the number of 'intruders' @xcite and automating the task of intruder detection @xcite . there is also work on topic diversity @xcite . however, there is a tradeoff between diversity and coherence @xcite , meaning high diversity for topic modelling is likely to be in conflict with thematic coherence, the main focus of the paper. moreover, we are ensuring semantic diversity of microblog clusters through our sampling strategy (see sec. 3.4). text generation metrics: tgms have been of great use in applications such as machine translation @xcite @xcite ), text summarisation @xcite and image captioning @xcite @xcite , where a machine generated response is evaluated against ground truth data constructed by human experts. recent advances in contextual language modeling outperform traditionally used bleu @xcite and rouge @xcite scores, which rely on surface-level n-gram overlap between the candidate and the reference. in our work, we hypothesise that metrics based on contextual embeddings can be used as a proxy for microblog cluster thematic coherence. specifically, we consider the following tgms: (a) bertscore is an automatic evaluation metric based on bert embeddings @xcite . the metric is tested for robustness on adversarial paraphrase classification. however, it is based on a greedy approach, where every reference token is linked to the most similar candidate token, leading to a time-performance trade-off. the harmonic mean f bert is chosen for our task due to its most consistent performance @xcite . (b) moverscore @xcite expands from the bertscore and generalises word mover distance @xcite by allowing soft (manyto-one) alignments. the task of measuring semantic similarity is tackled as an optimisation problem with the constraints given by n-gram weights computed in the corpus. in this paper, we adopt this metric for unigrams and bigrams as the preferred embedding granularity. (c) bleurt @xcite ) is a state-ofthe-art evaluation metric also stemming from the success of bert embeddings, carefully curated to compensate for problematic training data. its authors devised a novel pre-training scheme leveraging vast amounts of synthetic data generated through bert mask-filling, back-translation and word dropping. this allows bleurt to perform robustly in cases of scarce and imbalanced data. ## methodology notation we use @xmath0, ..., c n } to denote a set of clusters c i . each cluster c i is represented by the pair c @xmath1, w i ), where t i and w i represent the set of tweets and top-20 topic words of the dominant latent topic in c i , respectively. the task of identifying thematic coherence in microblog clusters is formalised as follows: given a set of clusters c, we seek to identify a metric function f : @xmath2.t. high values of f (c i ) correlate with human judgements for thematic coherence. here we present (a) the creation of a corpus of topic clusters of tweets c and (b) the annotation process for thematic coherence. (a) involves a clustering (sec. 3.2), a filtering (sec. 3.3) and a sampling step (sec. 3.4); (b) is described in (sec. 3.5). experiments to identify a suitable function f are in sec. 4. ## experiments our premise is that a pair of sentences scoring high in terms of tgms means that the sentences are semantically similar. when this happens across many sentences in a cluster then this denotes good cluster coherence. following @xcite , we consider three approaches to implementing and adapting tgms to the task of measuring thematic coherence. the differences between these methods consist of: (a) the choice of the set of tweet pairs s ⊂ t × t on which we apply the metrics and (b) the score aggregating function f (c) assigning coherence scores to clusters. the tgms employed in our study are bertscore @xcite , moverscore @xcite for both unigrams and bigrams and bleurt @xcite . we also employed a surface level metric based on cosine similarity distances between tf-idf representations foot_7 of tweets to judge the influence of word co-occurrences in coherence analysis. each approach has its own advantages and disadvantages, which are outlined below. ## conclusions and future work we have defined the task of creating topic-sensitive clusters of microblogs and evaluating their thematic coherence. to this effect we have investigated the efficacy of different metrics both from the topic modelling literature and text generation metrics tgms. we have found that tgms correlate much better with human judgement of thematic coherence compared to metrics employed in topic model evaluation. tgms maintain a robust performance across different time windows and are generalisable across several datasets. in future work we plan to use tgms in this way to identify thematically cluster annotation: bad random common keyword: 'oil' m'gonna have a nap, i feel like i've drank a gallon of like grease or oil or whatever bc i had fish&chips like 20 minutes ago check out our beautiful, nostalgic oil canvasses. these stunning images will take you back to a time when life... five years later, bottlenose dolphins are still suffering from bp oil disaster in the gulf. take action! once the gas and oil run out countries like suadia arabia and russia won't be able to get away with half the sh*t they can now ohhh this tea tree oil is burning my face off if the dinghies were painted pink would there be as much uproar about migrants drowning as the colour of a f**king bus?