Measuring What Counts: The Case of Rumour Stance Classification (2020)

Abstract: Stance classification can be a powerful tool for understanding whether and which users believe in online rumours. The task aims to automatically predict the stance of replies towards a given rumour, namely support, deny, question, or comment. Numerous methods have been proposed and their performance compared in the RumourEval shared tasks in 2017 and 2019. Results demonstrated that this is a challenging problem since naturally occurring rumour stance data is highly imbalanced. This paper specifically questions the evaluation metrics used in these shared tasks. We re-evaluate the systems submitted to the two RumourEval tasks and show that the two widely adopted metrics – accuracy and macro-F1 – are not robust for the four-class imbalanced task of rumour stance classification, as they wrongly favour systems with highly skewed accuracy towards the majority class. To overcome this problem, we propose new evaluation metrics for rumour stance detection. These are not only robust to imbalanced data but also score higher systems that are capable of recognising the two most informative minority classes (support and deny).

## introduction the automatic analysis of online rumours has emerged as an important and challenging natural language processing (nlp) task. rumours in social media can be defined as claims that cannot be verified as true or false at the time of posting (zubiaga et al., 2018). prior research (mendoza et al., 2010; kumar and carley, 2019) has shown that the stances of user replies are often a useful predictor of a rumour's likely veracity, specially in the case of false rumours that tend to receive a higher number of replies denying them (zubiaga et al., 2016). however, their automatic classification is far from trivial as demonstrated by the results of two shared tasks -rumoureval 2017 and 2019 (derczynski et al., 2017; gorrell et al., 2019). more specifically, sub-task a models rumour stance classification (rsc) as a four-class problem, where replies can: this paper re-evaluates the sub-task a results of rumoureval 2017 and 2019. @xcite it analyses the performance of the participating systems according to different evaluation metrics and shows that even macro-f 1, that is robust for evaluating binary classification on imbalanced datasets, fails to reliably evaluate the performance on rsc. this is particularly critical in rumoureval where not only is data imbalanced, but also two minority classes (deny and support) are the most important to classify well. based on prior research on imbalanced datasets in areas other that nlp (e.g. yijing et al. (2016) and elrahman and abraham (2013)), we propose four alternative metrics for evaluating rsc. these metrics change the systems ranking for rsc in rumoureval 2017 and 2019, rewarding systems with high performance on the minority classes. ## evaluation metrics for classification we define t @xmath0, t @xmath1, f @xmath2, where t p c (f p c ) is equivalent to the true (false) positives and t n c (f n c ) is equivalent to the true (false) negatives for a given class c. accuracy (acc) is the ratio between the number of correct predictions and the total number of predictions (n ): @xmath3, where c is the number of classes. acc only considers the values that were classified correctly, disregarding the mistakes. this is inadequate for imbalanced problems like rsc where, as shown in precision (p c ) and recall (r c ) p c is the ratio between the number of correctly predicted instances and all the predicted values for c: p @xmath4. r c is the ratio between correctly predicted instances and the number of instances that actually belongs to the class c: r @xmath5. macro-f β f β c score is defined as the harmonic mean of precision and recall, where the per-class score can be defined as: area under the roc curve receiver operating characteristic (roc) (fawcett, 2006) assesses the performance of classifiers considering the relation between r c and the false positive rate, defined as (per class): f p r @xmath6. since rsc consists of discrete classifications, roc charts for each c contain only one point regarding the coordinate (f p r c , r c ). area under the roc curve (au c) measures the area of the curve produced by the points in an roc space. in the discrete case, it measures the area of the polygon drawn by the segments connecting the vertices ((0, 0), (f p r c , r c ), (1, 1), (0, 1)). high au c scores are achieved when r (probability of detection) is maximised, while f p r (probability of false alarm) is minimised. we experiment with a weighted variation of au c: weighted macro-f β a variation of macro-f β, where each class also receives different weights, is also considered: we use β = 1 (p and r have the same importance) and β = 2 (r is more important). arguably, misclassifying denies and supports (f n d and f n s , respectively) is equivalent to ignore relevant information for debunking a rumour. since f n s negatively impact r, we hypothesise that β = 2 is more robust for the rsc case. our results show that the choice of evaluation metric has a significant impact on system ranking. in rumoureval 2017, the winning system based on acc was turing. however, figure 2 shows that this system classified all denies in-correctly, favouring the majority class (comment). when looking at the macro-f 1 score, turing is classified as fifth, whilst the winner is ecnu, followed by uwaterloo. both systems also perform very poorly on denies, classifying only 1% and 3% of them correctly. on the other hand, the four proposed metrics penalise these systems for these errors and rank higher those that perform better on classes other than the majority one. for example, the winner according to gm r, wf 1 and wf 2 is niletmrg that, according to figure 2 , shows higher accuracy on the deny, support and query classes, without considerably degraded performance on the majority class. wau c still favours the mama edha system which has very limited performance on the important deny class. as is evident from figure 2 , niletmrg is arguably the best system in predicting all classes: it has the highest accuracy for denies, and a sufficiently high accuracy for support, queries and comments. using the same criteria, the second best system should be iitp. the only two metrics that reflect this ranking are gm r and wf 2. in the case of wf 1, the second system is uwaterloo, which has a very low accuracy on the deny class. for rumoureval 2019, the best system according to macro-f 1 (the official metric) is blcu nlp, followed by but-fit. however, after analysing the confusion matrices in figure 3 , we can conclude that eventai is a more suitable model due to its high accuracy on support and deny. metrics gm r, wau c and wf 2 show eventai as the best system. finally, wshuyi is ranked as fourth according to gm r, wau c and wf 2, while it ranked seventh in terms of macro-f 1, behind systems like gwu and sinai-dl that fail to classify all deny instances. although wshuyi is clearly worse than eventai, blcu nlp and but-fit, it is arguably more reliable than systems that misclassify the large majority of denies. @xcite our analyses suggest that gm r and wf 2 are the most reliable for evaluating rsc tasks. ## weight selection in section 3, wau c, wf 1 and wf 2 have been obtained using empirically defined weights (w @xmath7.40, w @xmath8.15 and w @xmath9.05). these values reflect the key importance of the support and deny classes. although query is less important than the first two, it is nevertheless more informative than comment. previous work tried to adjust the learning weights in order to minimise the effect of the imbalanced data. garcía lozano et al. (2017) (mama edha), change the weights of their convolutional neural network (cnn) architecture, giving higher importance to support, deny and query classes, to better reflect their class distribution. @xcite ghanem et al. (2019) (upv) also change the weights in their logistic regression model in accordance with the data distribution criterion. @xcite nevertheless, these systems misclassify almost all deny instances. ## discussion this paper re-evaluated the systems that participated in the two editions of rumoureval task a (stance classification). we showed that the choice of evaluation metric for assessing the task has a significant impact on system rankings. the metrics proposed here are better suited to evaluating tasks with imbalanced data, since they do not favour the majority class. we also suggest variations of au c and macro-f β that give different weights for each class, which is desirable for scenarios where some classes are more important than others. the main lesson from this paper is that evaluation is an important aspect of nlp tasks and it needs to be done accordingly, after a careful consideration of the problem and the data available. in particular, we recommend that future work on rsc uses gm r and/or wf β (preferably β = 2) as evaluation metrics. best practices on evaluation rely on several metrics that can assess different aspects of quality. therefore, relying on several metrics is likely the best approach for rsc evaluation.