Masking and Transformer-based Models for Hyperpartisanship Detection in News (2021)

Abstract: Hyperpartisan news show an extreme manipulation of reality based on an underlying and extreme ideological orientation. Because of its harmful effects at reinforcing one’s bias and the posterior behavior of people, hyperpartisan news detection has become an important task for computational linguists. In this paper, we evaluate two different approaches to detect hyperpartisan news. First, a text masking technique that allows us to compare style vs. topic-related features in a different perspective from previous work. Second, the transformer-based models BERT, XLM-RoBERTa, and M-BERT, known for their ability to capture semantic and syntactic patterns in the same representation. Our results corroborate previous research on this task in that topic-related features yield better results than style-based ones, although they also highlight the relevance of using higher-length n-grams. Furthermore, they show that transformer-based models are more effective than traditional methods, but this at the cost of greater computational complexity and lack of transparency. Based on our experiments, we conclude that the beginning of the news show relevant information for the transformers at distinguishing effectively between left-wing, mainstream, and right-wing orientations.

## introduction media such as radio, tv channels, and newspapers control which information spreads and how it does it. the aim is often not only to inform readers but also to influence public opinion on specific topics from a hyperpartisan perspective. social media, in particular, have become the default channel for many people to access information and express ideas and opinions. the most relevant and positive effect is the democratization of information and knowledge but there are also undesired effects. one of them is that social media foster information bubbles: every user may end up receiving only the information that matches his/her personal biases, beliefs, tastes and points of view. because of this, social media are a breeding ground for the propagation of fake news: when a piece of news outrages us or matches our beliefs, we tend to share it without checking its veracity; and, on the other hand, content selection algorithms in social media give credit to this type of popularity because of the click-based economy on which their business are based. another harmful effect is that the relative anonymity of social networks facilitates the propagation of toxic, hate and exclusion messages. therefore, social media contribute to the misinformation and polarization of society, as we have recently witnessed in the last presidential elections in usa or the brexit referendum. clearly, the polarization of society and its underlying discourses are not limited to social media, but rather reflected also in political dynamics (e.g., like those found in the us congress @xcite ): even in this domain, however, social media can provide a useful signal to estimate partisanship @xcite . closely related to the concept of controversy and the "filter bubble effect" is the concept of bias @xcite , which refers to the presentation of information according to the standpoints or interests of the journalists and the news agencies. detecting bias is very important to help users to acquire balanced information. moreover, how a piece of information is reported has the capacity to evoke different sentiments in the audience, which may have large social implications (especially in very controversial topics such as terror attacks and religion issues). in this paper, we approach this very broad topic by focusing on the problem of detecting hyperpartisan news, namely news written with an extreme manipulation of the reality on the basis of an underlying, typically extreme, ideology. this problem has received little attention in the context of the automatic detection of fake news, despite the potential correlation between them. seminal work from @xcite presents a comparative style analysis of hyperpartisan news, evaluating features such as characters n-grams, stop words, partof-speech, readability scores, and ratios of quoted words and external links. the results indicate that a topic-based model outperforms a style-based one to separate the left, right and mainstream orientations. more recently, in @xcite , the features that participants used in semeval-2019 task 4 on hyperpartisan news detection have been summarized: n-grams, word embeddings, stylometry (e.g., punctuation and article structure), sentiment and emotion features, named entities, quotations, hyperlinks, and publication date. using the same dataset from semeval-2019, @xcite evaluated features like bag-of-words, bag-of-clusters, word embeddings and contextual character-based embeddings, pos n-grams, stylistic features and the sentiment; the authors found that dense document representations work better across domains and tasks than traditional sparse representations. finally, @xcite found effective to use personality information in hyperpartisan news detection after topic-based sub-sampling of the news training data. the datasets proposed in @xcite were manually labeled and the largest one was labeled in a semi-automated manner via distant supervision. instead of employing the datasets from @xcite , we build upon previous work and use the dataset from @xcite : this way we can investigate hyperpartisan-biased news (i.e., extremely one-sided) that have been manually fact-checked by journalists from buzzfeed, and contrast our results with what they achieved. the articles originated from 9 well-known political publishers, three each from the mainstream, the hyperpartisan left-wing, and the hyperpartisan right-wing. to detect hyperpartisanship, we aim to explore the trade-off between the performance of the models and the transparency of their results. taking this into account, we apply two approaches diametrically opposite to each other in the text classification state of the art. on the one hand, we use three transformer-based models, which have shown outstanding performance, but high complexity and lack of transparency. on the other hand, we use a masking-based model that requires fewer computational-resources and showed a good performance in related tasks such as authorship attribution @xcite . the masking technique transforms the original texts in a form where the textual structure is maintained, while letting the learning algorithm focus on the writing style or the topic-related information. this technique makes it possible for us to corroborate previous results that content matters more than style. moreover, we aim to find explainable predictions of hyperpartisanship with the attention mechanism of the transformer-based models. with this purpose, we expect to derive the explanation by investigating the scores of different features used to output the final prediction. based on this, we contrast the transparency of both approaches by comparing the relevant parts of the texts that they highlight. the rest of the paper is structured as follows. in section 2 we describe our method to hyperpartisan news detection based on masking. section 3 presents details on the dataset and the experimental setup. in section 4 we show the obtained results and discuss about them. finally, section 5 concludes with some directions for future work. ## experiments we used the buzzedfeed-webis fake news corpus 2016 collected by @xcite whose articles were labeled with respect to three political orientations: mainstream, left-wing, and right-wing (see ## results and discussion the last three rows of ## conclusions in this paper we presented initial experiments on the task of hyperpartisan news detection. in particular, we aimed to explore the trade-off between performance and transparency, and proposed a comparison of two different approaches. first, we explored the use of masking techniques to boost the performance of a lexicalized classifier. our results corroborate previous research on the importance of content features to detect extreme content: masking, in addition, shows the benefits of reducing data sparsity for this task comparing our results with the state of the art. we evaluated different values of the parameters and see that finally our baseline model, in which we extract character 5-grams without applying any masking process, achieves the better results. this seems to indicate a strong lexical overlap between different sources with the same orientation, which, in turn, calls for more challenging datasets and task formulations to encourage the development of models covering more subtle, i.e., implicit, forms of bias. future datasets could consider more topics and different time spans to avoid the models learn from the topic, rather than the target classes. second, we used three transformer-based models (bert, m-bert, and xlm-roberta) that are resource-hungrier than the masking technique, and achieved the highest results. we also presented some examples of how these models, through their attention scores, provide additional information about the relevant parts of the text for distinguishing their political orientation. considering the high effectiveness of these models, and that they only observe the first part of the news, we will evaluate as future work how necessary is to use all the news (and not only the beginning), e.g. with the transformer-xl model @xcite . moreover, we are motivated to take advantage of the attention scores to study in more detail the style used in hyperpartisan news in order to improve the predictions.