Rumour Detection in the Wild: A Browser Extension for T witter (2023)

Abstract: Rumour detection, particularly on social media, has gained popularity in recent years. The machine learning community has made significant contributions in investigating automatic methods to detect rumours on such platforms. However, these state-of-the-art (SoTA) models are often deployed by social media companies; ordinary end-users cannot leverage the solutions in the literature for their own rumour detection. To address this issue, we put forward a novel browser extension that allows these users to perform rumour detection on Twitter. Particularly, we leverage the performance from SoTA architectures, which has not been done previously. Initial results from a user study confirm that this browser extension provides benefit. Additionally, we examine the performance of our browser extension’s rumour detection model in a simulated deployment environment. Our results show that additional infrastructure for the browser extension is required to ensure its usability when deployed as a live service for Twitter users at scale.

## introduction the advent of social media has forever transformed the way in which we are able to communicate with one another. content-sharing has effectively been democratised with the removal of existing traditional barriers @xcite . indeed, this provides many benefits: for those individuals involved in a newsworthy event, for example, information can be shared in real-time. this allows for news to propagate faster, with the intention of informing the wider public and inciting action from relevant stakeholders. however, with the low barrier of entry to content production and dissemination on social media, the overall quality of information on these platforms has degraded @xcite . users of social media platforms are now able to, inadvertently or deliberately, publish misinformation and falsehoods @xcite . this has led to very damaging consequences for society in the past: @xcite have provided examples of these effects in the financial, political and social domains. with this in mind, the machine learning community has created automatic detection systems which are designed to identify misinformation on social media platforms. however, it is the case that companies who own and manage these platforms are the ones that implement and deploy their own misinformation detection services @xcite . as such, regular content-consumers do not have access to these services directly. this takes away the autonomy of an individual who wishes to perform rumour detection for themselves, for example. in this work, we present a google chrome browser extension, and the associated server architecture, (seen in figure 1 ) that addresses this problem as highlighted by @xcite . it allows regular twitter users to perform on-demand rumour detection for any tweet, whilst enhancing their experience through proving semantically-related news articles. ## related work considerable progress has been made in investigating methods to perform rumour detection on social media platforms. seminal papers in this discipline initially focussed on generating informative, handcrafted features for this task @xcite @xcite . stemming from these works, researchers refocussed their attention on temporal features. @xcite showed that these features are highly predictive of rumours as they described the periodic bursts that are typical for these phenomena. these new features now allowed researchers to capture how rumours change over time, which is particularly important for early rumour detection. @xcite identified rumours through modelling their lifecycle as a time series, whilst @xcite approached this through modelling the propagation pattern as a tree. furthermore, certain features were shown to have greater importance at different steps in a rumours' propagation @xcite . however, the collective flaw in these works is that the feature engineering processes are detailspecific, could introduce biases and are extremely laborious @xcite . as such, sota solutions turn to deep learning for the automatic feature representations, increased model complexity and subsequent increase in performance for rumour detection. @xcite implemented the same time series ideas from their earlier works @xcite with recurrent neural networks (rnns), later improved with attention @xcite , which leveraged the deep hidden representations that were learnt by the neural network. @xcite found that recursive neural networks (rvnns) were more performant than rnns as they are able to embed both content-based and propagation-based information due to their tree-like structure. this architecture was also improved with an attention mechanism @xcite . @xcite achieved sota performance on the twitter15 and twitter16 datasets for rumour detection on twitter through not only modelling the propagation properties of a rumour, but also the dispersion properties with their bi-directional graph convolutional network (bi-gcn). however, deploying said rumour detection models as a service has not been extensively researched in an academic setting. @xcite introduced tweetcred, a google chrome browser exten-sion that assigned a credibility score to each tweet on a user's feed using an svm-rank model using 45 handcrafted features. thilakarathna et al. (2020) created a browser extension for twitter, called veritas, that is able to detect fake news on the social media platform. most notably, their architecture involves a model trainer pipeline to ensure that their neural models are constantly up-to-date. @xcite explored a very similar tool that used a deep learning solution as the backbone to a browser extension focusing on clickbait detection. additionally, we note that news articles are a typical resource that one would use to determine the status of a rumour. these resources enhance the experience for the user, informing them of the context in which the tweet occurs. to this end, our work attempts to address the following flaws found in previous work. i) previous works that tackle rumour detection specifically do not leverage sota rumour detection models at the core of their service, which renders them outdated. ii) in certain cases, the rumour detection model is deployed on the user-side. we posit that with sota models, this will render the service inutile due to the computational complexity of the models @xcite . iii) none of the aforementioned works enhance the user's experience through recommending articles that are semantically related to the tweet in question. our browser extension will explore this addition. ## browser extension architecture when creating a browser extension, and the associated architecture, we wish for the following properties to be met. these are chosen to maximise the extension's ease-of-use, performance and informativeness. d1: twitter users should be able to use the service in-real time in concert with their ordinary browsing experience. d2: users of the service should not have to bear the computational load of the underlying architecture. d3: the detection model should be interchangeable, allowing for the browser extension to improve continually with the latest advancements in the field. d4: news articles that are semantically related to the source tweet should enhance the user's rumour detection experience. to meet the above desiderata, we put forward the browser extension architecture seen in figure 1 . furthermore, we designed a user-friendly interface in the form of a google chrome browser extension seen in figure 2 . a user will interact with the system as follows: once they identify a particular tweet on which they wish to perform rumour detection, the user opens the browser extension and clicks the "detect rumour" button seen in figure 2a (d1). once the button has been pressed, the tweet id is extracted from the url of the tweet and the client's web browser sends a post request to the web server (d2), with the specific tweet id as a parameter. ## datasets we make use of the twitter15 and twitter16 datasets @xcite to train our rumour detection model. the datasets comprise rumours linked to newsworthy events at specific time periods; the statistics of these can be found in edges between tweets represent their retweet or response relationships. a collection of tweets in a cascade describes an event, and is assigned a veracity tag (rumour (ur), non-rumour (nr), false rumour (fr) or true rumour (tr)) which were derived from cross-referencing rumour debunking websites. 5 rumour detection model evaluation ## latency experiments similar to the work done by @xcite , we wish to evaluate the browser extension's performance with respect to response time. this is calculated as the amount of time taken for our browser extension to respond to a particular rumour detection request. in our experiment, we measure the response time across 50 randomly selected newsworthy tweets of varying size. we view the cumulative distribution function (cdf) of the response times in figure 6 . observing figure 6 , we see that our browser extension is able to provide a response for 90% of requests in six seconds or less. this is an improvement over the 82% of requests in @xcite ; however, their experiment analysed the cdf across 5.4 million requests. due to time constraints, our sample size is significantly smaller. yet, we can make some initial comparisons between the two solutions. our work retrieves the entire tweet cascade for a tweet, and predicts the rumour status using the bi-gcn model. tweetcred, on the other hand calculates 45 handcrafted features based on the tweet itself, and uses svm-rank to assign a credibility score. given the fact that our approach uses far more information per tweet, and a more sophisticated and computationally intensive model foot_6 , this is still an encouraging result for our browser extension. unfortunately, the other browser extensions mentioned in section 2 do not report results for a response time experiment. as such, we are unable to compare our extension to other solutions on this particular axis. ## user study to determine whether the browser extension provides benefit to twitter users, we ran an anonymised user study. we asked 19 participants to perform five rumour detection tasks (rdts). in this context, a rdt is divided into two questions. each set of questions was asked on a preselected tweet that was newsworthy at that instance in time. this was done to ensure that the tweets used in the study were as similar as possible to the training data distribution on which the model was trained (see section 5.1 for out-of-distribution performance). after performing these tasks, we asked the users to comment on their overall experience using the browser extension. the user study was facilitated through the use of anonymised survey. the users accessed the server architecture through a remote google cloud platform server (gcp), and the browser extension itself from a shareable google chrome store link. the user study was approved by a research ethics board (see section 8). the results from the rdt and overall experience feedback are seen in figures 7 and 8 , respectively. see appendix a for full details on the user study. prior to performing rumour detection, we see that there is considerable disagreement amongst the annotators. particularly, we find that randolph's kappa is κ = 0.345 @xcite . these results confirm an assumed truth in the field: rumour t w e e t 1 t w e e t 2 t w e e t 3 t w e e t 4 t w e e t 5 0 2 4 6 8 10 12 14 16 18 frequency did you have reason to doubt the tweet a priori? yes no t w e e t 1 t w e e t 2 t w e e t 3 t w e e t 4 t w e e t 5 did the browser extension support your experience? yes no figure 7: results from the rdt in the user study. percentage of responses (%) i found this browser extension to be very useful. i would recommend this browser extension to my friends and family. i would use this browser extension again in my personal time. 5.26 5.26 10.53 15.79 5.26 5.26 42.11 31.58 52.63 36.84 57.89 31.58 strongly disagree disagree neither agree nor disagree agree strongly agree detection, a highly subjective task, is difficult for humans as the labels are not necessarily objective @xcite . this further motivates the need for assistive tools such as this browser extension. contrastingly, we see that, generally, annotators found that the browser extension supported their rumour detection experience. κ = 0.443 which supports the fair agreement on the browser extension's positive impact on their rumour detection experience. an interesting result is that this kappa score is similar to the user agreement on the credibility score (43%) obtained for tweetcred @xcite . whilst the questions posed to the users are different, these results show that there is some benefit to be gained through using additional rumour detection tools. however, there needs to be additional measures put into place to make users more confident in the tool's performance, which would lead to higher user agreement (see section 8). furthermore, we asked the users to rate their agreement to three questions on a five-point likert scale. observing figure 8 , we see that the feedback for the browser extension is generally positive. 78.95% of the participants in the user study agreed or strongly agreed with finding the browser extension to be useful. however, we see that there was a small portion of users who either disagreed or were ambivalent to the three statements. this study did not require the participant to be a twitter user. as such, the study could have attracted partic-ipants who: do not use twitter frequently, do not use twitter as a news source, or those users that do not have a twitter account at all. these users would not see the need to have access to a tool such as this browser extension as they would have no use for it personally. ## conclusion and future work in this work, we put forward a novel browser extension that allows twitter users to perform rumour detection, leveraging the performance from sota models. our work shows that this tool provides benefit to those twitter users wanting autonomy over their rumour detection. however, we note that our work is merely the first iteration in a series of deployments. future work could explore additional mechanisms to allow the browser extension to cope with ood data. online retraining @xcite has been shown to be effective in minimising the effects of concept drift; this is similar to the trainer pipeline in veritas @xcite . furthermore, @xcite show a more sophisticated paradigm with their continual learning approach. additionally, we could extend the browser extension's functionality through allowing its users to submit examples of tweets they believe to be (non-)rumours (with evidence) to some community-moderated data store. this process could be used to create more up-todate datasets for rumour detection.