Incorporating Count-Based Features into Pre-Trained Models for Improved Stance Detection (2020)

Abstract: The explosive growth and popularity of Social Media has revolutionised the way we communicate and collaborate. Unfortunately, this same ease of accessing and sharing information has led to an explosion of misinformation and propaganda. Given that stance detection can significantly aid in veracity prediction, this work focuses on boosting automated stance detection, a task on which pre-trained models have been extremely successful on, as on several other tasks. This work shows that the task of stance detection can benefit from feature based information, especially on certain under performing classes, however, integrating such features into pre-trained models using ensembling is challenging. We propose a novel architecture for integrating features with pre-trained models that address these challenges and test our method on the RumourEval 2019 dataset. This method achieves state-of-the-art results with an F1-score of 63.94 on the test set.

## introduction and motivation in today's world, it is hard to imagine life without the internet and social media. the exponential growth of social media platforms over the past decade has provided people with an extremely convenient medium to access, consume and disseminate information. social media has also become the primary source of news due to its ability to allow users to follow real-time updates @xcite . unfortunately, this has also led to the spread of misinformation and propaganda. a recent example is that of covid-19 which saw a plethora of misinformation online @xcite . social media users often have trouble identifying credible resources on various platforms and are frequently guided towards misinformation, causing users to trust misinformation. with about 200 billion tweets posted every day @xcite , manually processing this information to detect rumour can be extremely tedious, time consuming, prone to human error and bias and most importantly impractical. thus, it is essential to develop efficient automated methods that help in detecting and tracking misinformation online to aid in the fight against misinformation. there are four primary components to rumour determination: rumour detection, rumour tracking, stance classification, and veracity classification @xcite . stance classification aims at studying the stance of a reply post (referred to as an 'opinion') on the source post (referred to as the 'target' -to what the opinion is subjected). studies have shown that posts containing false information have a notably higher number of 'denying' replies than the ones that do not @xcite . hence, stance classification can play an important role in assessing the veracity of a rumour. given that stance classification can strongly aid in determining the veracity of a post, it is an important if challenging problem to solve. it remains challenging because posts are often sarcastic or witty, while also questioning the validity of the original poster's assumptions or evidence @xcite . ## related work the effectiveness of stance detection in identifying and tracking misinformation online has led to significant research in this area. this section provides an overview of work related to this study. early work on studying perspectives such as that by @xcite use naïve bayes and svm based classifiers to show that word usage alone can help in determining the point of view of a document or sentence. in a similar study, @xcite go beyond word usage and exploit the lexical patterns used by people while expressing an opinion. @xcite showed that a unigram based model outperformed a sentiment-based model for determining stance thus reinforcing the importance of words in detecting stance. this shows that the stance of an opinion could be related to the kind of words used to express it. although working on a different problem set (focused on binary classification) and using different datasets, the above-mentioned studies are relevant to this work because they emphasise on the importance of words used to express an opinion to determine their stance. @xcite hypothesise the presence of certain words in the reply text or the opinion as an indicative measure of its stance. they use these and some tweet specific features like the word count of the tweet to train on a gradient boosting classifier and show that stance classifiers could benefit from these topic independent features. @xcite explore a novel set of features and claim that simpler classifiers with profuse feature knowledge can outperform several complex machinelearning techniques in stance detection. @xcite use a multi layer perceptron (mlp) having a single hidden layer with term frequency and tf-idf vectors as inputs to perform at par with several complex models. these works lend weight to our hypothesis that handcrafted and count-based features can significantly aid in the task of stance detection. recent work in stance classification shows extensive use of deep-learning models to achieve state-ofthe-art results. @xcite and @xcite who worked on the same task and dataset as this work, use the bert and gpt architectures respectively for stance classification. @xcite use an ensemble model wherein they combine outputs from several bert models to increase the f1-score. their system achieves a significantly high f1-score without the use of any hand-crafted features. @xcite , who achieved state of the art results on this dataset prior to this work, use an inference chain based system that was fine-tuned on gpt. they use handcrafted features such as the presence of question marks, urls, positive and negative words, etc. and leverage the structure of a conversation thread to achieve state-of-the-art results in stance classification. ## experimental and model design this section describes initial exploratory experiments and the process behind building a model that address the two obstacles to integrating features into pre-trained models mentioned in section 1.1, namely: a) the significant difference in the number of epochs required by pre-trained models and feature based models, and b) the fact that using raw features might contain too much noise in an ensemble, the outputs alone might contain too little information, and dimensionality reduction methods result in the loss of task specific information. ## results and analysis this section presents an overview of our experiments and findings. we compare our results with prior state-of-the-art work on stance classification which makes use of this data set. we also perform an error analysis of our proposed model and report findings.