Revisiting Rumour Stance Classification: Dealing with Imbalanced Data (2020)

Abstract: Correctly classifying stances of replies can be significantly helpful for the automatic detection and classification of online rumours. One major challenge is that there are considerably more non-relevant replies (comments) than informative ones (supports and denies), making the task highly imbalanced. In this paper we revisit the task of rumour stance classification, aiming to improve the performance over the informative minority classes. We experiment with traditional methods for imbalanced data treatment with feature- and BERT-based classifiers. Our models outperform all systems in RumourEval 2017 shared task and rank second in RumourEval 2019.

## introduction a key step in the task of automatically analysing rumour veracity is to analyse the view of other users on a particular rumour @xcite , i.e. the stance of its replies. rumoureval 2017 and 2019 @xcite are shared tasks that provide tree-structured conversation threads consisting of tweets directly or indirectly replying to a rumourous tweet and aim to label the stance of these replies towards the rumour (task a). specifically, it is framed as a four-class classification problem: support, deny, query, and comment (sdqc). supports and denies are arguably the most informative stances for rumour verification @xcite , while comments are considered the least useful. however, the data for this task is highly imbalanced with supports and denies corresponding, respectively, to 18% and 7% of instances in the rumoureval 2017, while comments are 66%. 1 systems submitted for rumoureval 2017 task a, evaluated in terms of accuracy, have a high performance for the majority class (comments), whilst the minority classes are under-performed. mama edha @xcite adjusts the weights of the four labels, while only correctly classifying 1% denies and 37% supports. ecnu proposes a two-step classifier, however, only 1% of denies and 28% of supports are accurately predicted @xcite . iitp over-samples the underrepresented classes @xcite , although only 12% denies and 44% supports are recognised. the winner, turing @xcite , is unable to identify any denies in the test data. in rumoureval 2019, with macro-f 1 as evaluation metric, eventai @xcite (third place) achieves 55% and 79% of correct supports and denies, respectively. ranked first, blcu nlp @xcite increases the supports and denies with external similar datasets. however, the expanded data is still skewed towards comments, and 38% supports and 51% denies are accurately predicted. upv @xcite set different weights for each class, but 72% supports and 91% denies are mis-classified. despite gwu @xcite designing a rule-based model to help predict the instances of minority classes, their system does not correctly classify any denies. other work @xcite @xcite that also experiments with rumoureval or pheme datasets does not consider imbalanced data approaches, and under-performs in support and deny classes. in this paper, we experiment with well-known techniques for dealing with data imbalanced problems (e.g. smote @xcite ). we create feature-based models and a bert-based model @xcite . results show that our models not only have higher performance for the minority classes but also have higher overall performance (in terms of macro-f 1 and geometric mean recall -gmr) than all systems submitted to rumoureval 2017 and all but one system submitted to rumoureval 2019. foot_0 2 resampling mechanisms and threshold-moving random under-or over-sampling (rus or ros) rus randomly discards samples in the majority class so that the class proportions can be balanced. generally, it is computationally more efficient than over-sampling because it reduces the training data, although it may lead to under-fitting. ros re-balances the class proportions through randomly replicating samples in the minority classes. however, these replications can increase the possibility of over-fitting @xcite . ## conclusion and future work we experiment with traditional imbalanced data techniques for the task of rumour stance classification and show that: (i) our models are capable of outperforming all systems in rumoureval 2017 and all but one system in rumoureval 2019 in terms of both macro-f 1 and gmr scores, and (ii) a more in-depth evaluation is needed in order to correctly assess this task. further improvements may be achieved by employing model-based imbalanced data techniques (e.g. by setting different weights for each class during training), which is left as future work.