Evaluating Unsupervised Representation Learning for Detecting Stances of Fake News (2020)

Abstract: Our goal is to evaluate the usefulness of unsupervised representation learning techniques for detecting stances of Fake News. Therefore we examine several pre-trained language models with respect to their performance on two Fake News related data sets, both consisting of instances with a headline, an associated news article and the stance of the article towards the respective headline. Specifically, the aim is to understand how much hyperparameter tuning is necessary when fine-tuning the pre-trained architectures, how well transfer learning works in this specific case of stance detection and how sensitive the models are to changes in hyperparameters like batch size, learning rate (schedule), sequence length as well as the freezing technique. The results indicate that the computationally more expensive autoregression approach of XLNet (Yanget al., 2019) is outperformed by BERT-based models, notably by RoBERTa (Liu et al., 2019).While the learning rate seems to be the most important hyperparameter, experiments with different freezing techniques indicate that all evaluated architectures had already learned powerful language representations that pose a good starting point for fine-tuning them.

## introduction with the rise of social media, exchange of opinions and news happens faster than ever. news circulation is therefore less and less bound to traditional print journalism that usually requires extensive research, fact checking and accurate coverage in order to be a reliable news resource. it is relatively easy to share opinions that are either not supported by researched facts or simply wrong. in the worst case a large amount of people can be targeted by propaganda in order to shift societal discussions in favor of a wanted agenda. human resources are limited to identify such fake news, since they cover a wide range of topics and linguistic writing styles @xcite . automated fake news detection (fnd) has therefore proven to be an important challenge for nlp researchers in recent years. to this day, a variety of approaches dealing with fnd exists @xcite . in 2017, the fake news challenge stage 1 was introduced which tackles fnd as a stance detection task @xcite , where the idea is to determine the stance of a news article to a given headline @xcite . we now evaluate the five models bert @xcite , roberta @xcite , distilbert @xcite , albert @xcite and xlnet @xcite which were developed and enhanced recently. all architectures were pretrained on large unlabeled corpora using a self-supervised objective and can be finetuned on a desired task at hand. our main focus is to evaluate the necessity of hyperparameter tuning as well as the general performance. 1 besides this, it is of special interest to examine the differences between auto-encoding (bert-based models) and auto-regressive architectures (xlnet). in this context, the term fake news is defined as a text piece that is verifiably wrong and spread with a malicious intention. in doing so, other media sources such as video, images or audio are excluded. since the intention has to be malicious all sorts of entertainment related false news such as hoaxes and april fools are excluded. the definition is similar to the narrow definition that @xcite undertake. in 2017, the fake news challenge stage 1 (fnc-1) was published. organizers from industry and academia created an online challenge accessible via http://www.fakenewschallenge.org/ @xcite . the fnc-1 is conceptualized as an important pre-step in identifying fake news and exploring how artificial intelligence tools can be leveraged in combatting them. given a certain claim about a topic, what are different news agencies reporting about this claim? if most news agencies agree with a claim, this can be interpreted as an indicator of the truthfulness of the claim. on the contrary, if a lot of news disagree with the claim, the claim is likely fake news. statistically speaking this idea is translated into a stance detection task with the claim being treated as a headline and the stance of the article body being either agree, disagree, discuss or unrelated. fnd is thus treated as a classification task with four categories which are interpreted as the stance of an article body towards a given headline claim. along with a baseline model, a training and test set was published. according to @xcite , 50 teams participated in the challenge. ## related work in general, fnd can be interpreted as a binary classification task where a text document is classified as either fake news or no fake news or as a multi-class problem often with ordinal labels. one major distinction in different classification algorithms can be drawn in the consideration of auxiliary information. most contributions take a purely feature-oriented approach, while others try to incorporate information revolving around spreading fake news and the respective dissemination process. the feature-based approach focuses on extracting relevant linguistic peculiarities associated with fake news. typical examples are characters n-grams, words or a measure of the readability and syntax of an article body @xcite . other authors included the average word length, counts of exclamation marks or the sentiment of an article @xcite . after finding appropriate features, traditional machine learning algorithms as well as (deep) neural networks are proposed to classify the text instance given the extracted features. other feature-based approaches directly rely on deep learning, using neural networks for learning good representations of the text input by a stack of hidden layers which is then fed in a last classification layer. this approach is used by @xcite who simultaneously train a cnn on text and image data to classify fake entities and by @xcite who implement a two-step approach of using supervised and unsupervised learning with a cnn as well. but despite the fact that feature-based approaches are fairly popular within the fnd research, @xcite argue that they are not sufficient. approaches that use auxiliary information e.g. revolve around modeling the dissemination process of fake news by incorporating spatio-temporal information about users who like, share or publish (potential) fake news @xcite . the fnc-1 organizers use a feature-based approach by proposing a baseline model that extracts various features from the headlines and article bodies. the approach of specifically modeling the relationship between a headline and a respective article body was also exploited by @xcite in the context of clickbait detection while the fnc-1 takes this approach as a pre-step to fnd. ## conclusion the overall conclusion that can be drawn from this work is how powerful and strong the evaluated architectures are for the examined task related to fake news detection. even with minimal hyperparameter tuning and only finetuning for 3 epochs, the models already performed considerably well on both data sets. with respect to the considered hyperparameters the two most important conclusions are to not only finetune the classification and pooling layers that are stacked on top of the pretrained models foot_3 . secondly, the most important hyperparameter is the learning rate. the recommendation of @xcite to focus on the learning rate, when one has only time to address one single hyperparameter can be confirmed. at last, the models are relatively robust with respect to the learning rate schedule, the batch size, as long as it is adjusted to the learning rate and to a certain degree also the sequence length. furthermore, the excessive pretraining approach of roberta can outperform the permutation language model objective of xlnet. fake news detection itself is often time treated as a inherently binary task foot_4 (i.e. a document either contains fake news or it does not). the underlying truth, however, might be more complicated, since most of the times a document can not unambiguously be assigned to either of these two classes. this is why our experiments on the fnc-1 data sets, which comprehend fake news detection as a multi-class classification problem, might be more beneficial for this field of research compared to experiments on data sets with a binary target. we would not a go as far as concluding that these findings are without further ado transferable to other fake news related data sets using different tasks or label sets, but nevertheless think that our findings can serve as starting points for further experiments in related fields.