It’s about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits (2023)

Abstract: New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Consequently, models trained on random splits may not perform well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor detection benchmarks considering chronological instead of random splits. Our experimental results show that the use of random splits can significantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should always be evaluated using chronological splits for minimizing topical overlaps.

## introduction unverified false rumors can spread faster than news from mainstream media, and often can disrupt the democratic process and increase hate speech @xcite . automatic detection of rumors is an important task in computational social science, as it helps prevent the spread of false rumors at an early stage @xcite @xcite . current rumor detection approaches typically rely on existing annotated benchmarks consisting of social media data, e.g., twitter 15 @xcite , twitter 16 @xcite , weibo @xcite , and pheme @xcite that cover a wide range of time periods. these benchmarks use random splits for train, development and test sets which entail some topical overlap among them (see dataset id post label leven twitter 15 407231* r.i.p to the driver who died with paul walker that no one cares about because he wasn't famous. rumor 3 407236* r.i.p to the driver that died with paul walker that no one cares about because he wasn't famous. rumor twitter 16 594687* the kissing islands, greenland. url non-rumor 0 604628* the kissing islands, greenland. url non-rumor pheme 498483* happening now in #ferguson url non-rumor 9 499402* right now in #ferguson url non-rumor weibo 349863* 【喝易拉罐一定要吸管】一妇女喝了罐饮料，被送进医院，离开了世界。 研究显示罐上面的 毒菌很多 请转给你关心的朋友。 translation: please forward to your friends you care about. rumor 10 350023* 【喝易拉罐一定要吸管】一妇女喝了罐饮料，被送进医院，离开了世界。 研究显示罐上面的 毒菌很多！！这些你知道么 translation: do you know about this? rumor using random splits also results into posts with almost identical textual content shared during the same period. in this paper, we design a battery of controlled experiments to explore the hypothesis that whether temporality affects the predictive performance of rumor classifiers. to this end, we re-evaluate models on popular rumor detection benchmarks using chronological data splits i.e., by training the model with earlier posts and evaluating the model performance with the latest posts. results show that the performance of rumor detection approaches trained with random data splits is significantly overestimated than chronological splits due to temporal concept drift. this suggests that rumor detection approaches should be evaluated with chronological data for real-world applications, i.e., to automatically detect emerging rumors. ## results random splits vs. chronological splits we also note that the effect of temporality varies in datasets of different size. for both data splitting strategies, we observe that the difference in performance is 50% higher for the two datasets with hundreds of posts (e.g., twitter 15 and twitter 16) and around 10% in ones with thousands of posts (e.g., pheme and weibo). for rumor detection tasks, temporality may have a greater impact on small-scale benchmarks than on large-scale benchmarks. for twitter 16 and weibo, the use of stratified chronological splits demonstrates significant performance drops compared to random splits due to the temporal concept drift. for chronological splits, we observe that pretrained language models (i.e., bert and bert+) significantly outperform (t-test, @xmath1.05) logistic regression in all benchmarks. this is due to the fact that bert-style models (i) outperform simpler linear models by a large margin in various nlp tasks @xcite ; and (ii) have been trained after the development of these four benchmarks implying some information leakage. ## error analysis finally, we perform an error analysis to further investigate the type of errors made by bert us- ## conclusion we have shed light on the impact of temporal drift on computational rumor detection. results from our controlled experiments show that the use of chronological splits causes substantially drops in predictive performance across widely-used rumor 3 we set the threshold value to 20.