An Incremental Clustering Baseline for Event Detection on T witter (2024)

Abstract: enter abstract here

## introduction with the development of social media, the ability to recognize events in streams of short texts-particularly tweets-has become increasingly important. this process, called event recognition, involves identifying significant occurrences within large volumes of data, posing various challenges. a key component of this task is defining a clear and operational concept of what qualifies as an event. in this paper, we will use a working definition of event, as proposed by @xcite . the authors propose in fact a double definition: "definition 1: an event is a significant thing that happens at some specific time and place". this needs to be completed by the definition of what 'significant' means, so they add: "definition 2: something is significant if it may be discussed in the media. for example you may read a news article or watch a news report about it". because this definition has been used to build other corpora, it can be considered functional. as a result, corpora created with this definition should be comparable, with different annotators likely producing similar outcomes. one of the main challenges in event recognition is then being able to cluster different texts that refer to the same event. this difficulty arises from the wide range of expressions used to describe similar events. different sources and users may refer to the same event using different expressions, making it essential for recognition systems to account for synonymy, paraphrasing, and other linguistic variations. moreover, the temporal dimension is also a critical parameter in event recognition. the timing of events and the sequence in which they are reported can significantly impact the interpretation and relevance of the information extracted. another challenge in event detection on social networks is the sheer volume of messages posted on these platforms: an effective algorithm must be capable of processing millions of tweets within a reasonable time frame. many studies propose computationally intensive methodologies that are impractical for many real-world applications. therefore, research in this field needs to establish baselines on publicly accessible datasets that are both performant and time-efficient. the primary objective of this paper is thus to establish a performance standard for event detection while maintaining an acceptable level of computational complexity. our approach involves the use of an incremental clustering algorithm enhanced by recent advancements in sentence embeddings. specifically, we build upon the incremental clustering algorithm introduced by @xcite in their dataset publication. while effective at the time, their approach relied on lexical descriptions that may now be outdated due to the development of new word embedding techniques, particularly those stemming from recent large language models based on the transformer architecture @xcite . in our study, we utilize sentence-bert @xcite , a model that is especially noteworthy for its ability to encode entire sentences from individual word encodings. the structure of the paper is as follows: first, we will review recent work in the domain. next, we will detail our method and experiments. finally, we will present and discuss our results, concluding with a broader discussion. we conduct experiments on two large public twitter datasets to demonstrate the state of the art performance, efficiency, and robustness of this method (note that our code is publicly accessible foot_0 ). we then aim to compare our results with previous studies in the field, specifically those by @xcite and @xcite . by leveraging these advanced sentence embeddings, we demonstrate that our implementation surpasses more recent and complex approaches in both time-efficiency and the quality of detected events. these short-text representations provide a sophisticated understanding of language and context, allowing for more accurate and nuanced event recognition. ## methodology when working with social media data, one needs to consider both the textual similarity of the documents and their temporal proximity to avoid grouping together tweets posted at significantly different times. since the number of events is not known in advance, the chosen algorithm does not require the number of events given a priori. following the method by @xcite , we use an incremental clustering algorithm derived from the topic detection and tracking @xcite initiative. algorithm. this mini-batch first story detection (fsd) algorithm works as follows: documents are vectorized (we develop embedding methods in the subsequent section), sorted chronologically, and processed in batches of b documents. each new batch is compared to a window of w previous documents in terms of cosine distance. for each batch document, if the distance to its nearest neighbor is smaller than a threshold t, it joins the same cluster as its nearest neighbor. otherwise, the document joins a new cluster. the procedure is detailed in algorithm 1, where δ denotes the cosine distance. short-text embeddings. in the work published by @xcite , the best performing algorithm 1: "mini-batch" fsd input: threshold t, window size w, batch size b, corpus @xmath0. . . d n-1 } of n documents in chronological order output :a list t of cluster ids for each document time complexity. the time complexity of the fsd algorithm is o(nw) (with n the number of documents in the collection and w the number of documents in the time window), since each document in the corpus is compared only with the last w documents in chronological order. in practice, when using the "mini-batch" fsd, computation time is inversely proportional to batch size, as illustrated in figure 1 . ## experiments baselines. we compare our results (fsd-sbert) with hisevent 2 (he), the most recent paper on event detection: cao et al.'s (2024) work on the partition of a graphical neural network representation of tweets using structural entropy minimization. we also evaluate the performance improvement achieved by using sentence transformers in comparison to the tf-idf vectors used in (tw) foot_2 by @xcite . ## results and discussion performance. we observe that @xcite mini-batch fsd algorithm combined with sentence transformers pre-trained on large text corpora consistently outperforms the baselines on both datasets. the comparison between hisevent (he) and twembeddings (tw) seems to indicate that the mini-batch first story detection algorithm, even used with a simple tf-idf representation of tweets, is still a strong baseline, since its performance is comparable (and even superior on the french dataset) to hisevent when using ami as the indicator, though it is inferior when evaluated with ari. it is important to note that @xcite also use sentence transformers as a baseline in their article, with a different clustering algorithm (kmeans). their results are represented as the first column in figure 2 , with the exact sbert models they have used ("all-minilm-l6-v2" foot_5 for english and "distiluse-base-multilingual-cased-v1" foot_6 for french). our experiments show that the type of sbert model has an effect on performance: as shown on figure 2 b, the "multilingual" model is less efficient for french than the language-specific "camembert" model. nevertheless, regardless of the model used, the fsd algorithm (see the last two columns) is much more efficient than the k-means for both datasets. this gap is explained by the fact that the fsd algorithm is able to take into account the temporality of tweets (by applying a sliding time window when searching for nearest neighbors) unlike the k-means. moreover, fsd seems to be robust to changes in sbert models without the need to adapt the parameters: on figure 2 , when using fsd, the same threshold @xmath1.55 is used for all sbert models. this common threshold explains the small difference between the values in the experiments shown on figure 1 were run on a notebook pc with 32gb of ram and and 8 2.4ghz cpus. note that these tests do not take into account the encoding of the tweets using sentence transformers, since we computed the embeddings only once on a gpu server and then stored them to be re-used for further experiments on a notebook computer without gpu. it took 65 seconds using a nvidia rtx a4500 gpu to encode the event2012 corpus, and 240 seconds to encode the event2018 corpus. resources. we observed that executing hisevent on the entire event2012 dataset required substantial memory resources, exceeding 62 gb of ram. in contrast, fsd operates with significantly lower memory requirements (less than 32gb of ram). limitations. twitter has been an invaluable resource for research on social media and real-time data streams. however, this is no longer possible due to the platform's api restrictions. nevertheless, we believe this study remains relevant, as other data streams and social networks continue to produce valuable data, and event recognition continues to be a crucial task. another limitation related to the mini-batch fsd algorithm is the need to pre-determine the hyperparameter t. however, the consistency of the results with the same t value across several sbert models (see figure 2 ) suggests that this threshold (@xmath2.55) could be appliced to other sentence transformers models pre-trained on corpora in different languages. finally, a potential improvement for this method would be to better account for the nested nature of events in public discourse: for instance, a major political event might consist of numerous smaller sub-events, such as speeches, protests, and negotiations (for example the yellow vest protest in france lasted several months, with protests every week, discussions with the government, thousands of declarations, actors and reactions @xcite . each of these sub-events can be reported separately (or not) in different messages. this layered structure would ideally necessitate more sophisticated models capable of capturing and integrating these various components to provide a coherent and comprehensive understanding of the overall event. ## conclusion in this study, we aimed to investigate the performance of incremental clustering combined with sentence transformers models for automatically detecting events in a stream of tweets. our results demonstrated that applying the mini-batch fsd algorithm to sbert representations significantly improves event detection performance on twitter. we suggest that future research in this area should adopt this straightforward approach as a baseline for deploying more complex algorithms.