Active Learning for Rumor Identification on Social Media (2021)

Abstract: Social media has emerged as a key channel for seeking information. Online users spend several hours reading, posting, and searching for news on microblogging platforms daily. However, this could act as a double-edged sword especially when not all information online is reliable. Moreover, the inherently unmoderated nature of social media renders identifying unverified information ever more challenging. Most of the existing approaches for rumor tracking are not scalable because of their dependency on a significant amount of labeled data. In this work, we investigate this problem from different angles. We design an Active-Transfer Learning (ATL) strategy to identify rumors with a limited amount of annotated data. We go beyond that and investigate the impact of leveraging various machine learning approaches in addition to different contextual representations. We discuss the impact of multiple classifiers on a limited amount of annotated data followed by an interactive approach to gradually update the models by adding the least certain samples (LCS) from the pool of unlabeled data. Our proposed Active Learning (AL) strategy achieves faster convergence in terms of the F-score while requiring fewer annotated samples (42% of the whole dataset for the best model).

## introduction rumor detection in social networks is the task of identifying if a post's remark is unverifiable. this detection can help stop the spread of misinformation/dis-information that could potentially cause harm and distress. when a rumor about a subject emerges, there are thousands of posts shared about that subject. @xcite show that having abundant in-domain labeled data can significantly impact the accuracy of the rumor detection model on tweets by more than 30% improvement. this also points to the impact of outof-domain/topic training on rumor detection per-formance. however in a real world scenarios for rumor detection, in domain human-annotated data is typically missing in early stages of rumor propagation, resulting in mediocre accuracy levels for such models. a viable solution for this problem would ideally be a framework that yields decent accuracy despite the absence of in-domain manually annotated training data. to this end, this paper proposes a semi-supervised framework based atl for rumor detection in social media, specifically for twitter data. there are three main variables for the proposed framework: the representation of the tweets, the estimator, and the active learning strategy. other experimental variables will be discussed in the following sections. as we evaluate all the different variables, we observe that tweet-bert, linear regression and least confidence strategy yield comparable results as non-active learning methods yet with a fraction of human-annotated data needed in non-active learning based methods. further for robustness of our proposed models, we experiment with using an exploration method by choosing some random queries in each loop to prevent the model from overfitting. we also reach an approximation of the minimum labeled data needed for a decent classification in this task with the proposed method. ## experimental setup we compare each experiment with least confidence (lc) and random strategy (rnd) leveraging different representations, and learning algorithms. in random strategy, data samples are chosen uniformly random at each iteration. to mitigate the effect of randomness in both strategies, training algorithms and data splits, for each experiment, we randomly split the dataset into two sections, first one for initial and pool data and the second one as test data. we do this 5 times, and average the results (in a cross validation type evaluation strategy). for each of these 5 runs, the splits are the same among different experiments. at each iteration, multi-layer perceptron (mlp) is retrained on the batch of data annotated in the current iteration since training from scratch would be computationally expensive. however, other models are trained from scratch on data that was obtained in the current and previous iterations. in all experiments, we train the models on 20 randomly chosen samples as the initial dataset and query for 50 samples at each iteration, i.e. batch @xmath0. after examining all the models and representation settings with lc and rnd strategy, the best setting is utilized for further experiments leveraging other sampling strategies such as query batched committee (qbc), epsilon-greedy (eg), and ranked batch (batch-lc). ## error analysis ## conclusion & future directions we proposed an active-transfer learning framework for the rumor detection task. in our proposed framework, we examined different word representations, estimators and active learning strategies. more than 300 experimental setups were run and each setup was fine tuned to yield the best results. our experiments indicate multiple new findings: 1. the approximate minimum number of labeled in-domain data needed for a decent rumor detection model with our proposed method is around 200; 2. in-genre pretrained (contextualized) lms have the biggest impact on model performance; 3. we investigate and empirically show how epsilongreedy inspired methods that joins randomness and uncertainty in query selection could prevent the model from over-fitting; and, 4. we also showed that naive use of tweets relating to other topics can degrade the performance (the zero shot setting). although the method proposed in this paper did not show improvement in using data from other topics, information from different topics can be exploited by incorporating other techniques such as weighting the data samples or meta-learning fewshot domain adaptation. diverse initial datasets may yield an initial model with better uncertainty scores and earlier convergence. the next step for this method would be incorporating metadata such as reply stances, user information, network propagation information, etc. finally, another method that could improve our proposed model would be using an ensemble of different representations and different models to generalize better.