Predicting Political Orientation in News with Latent Discourse Structure to Improve Bias Understanding (2022)

Abstract: With the growing number of information sources, the problem of media bias becomes worrying for a democratic society. This paper explores the task of predicting the political orientation of news articles, with a goal of analyzing how bias is expressed. We demonstrate that integrating rhetorical dimensions via latent structures over sub-sentential discourse units allows for large improvements, with a +7.4 points difference between the base LSTM model and its discourse-based version, and +3 points improvement over the previous BERT-based state-of-the-art model. We also argue that this gives a new relevant handle for analyzing political bias in news articles.

## introduction misinformation is a major threat on modern democracy, influencing political agendas in an arguably unfair way, through multiple sources that are more or less transparent in their orientations. biased media can influence public opinion by selecting reported facts and angles, oriented presentation of events, with a proven impact, e.g. on electoral behaviours @xcite or public health @xcite . the automatic identification of such biases can thus help more transparent and democratic sharing of information, and the understanding of its typical expression. the study of bias has generated a lot of interest in political sciences with some emphasis on its linguistics aspects @xcite , which also gave rise to numerous studies on automating bias detection @xcite . nlp approaches mostly rely on lexical information @xcite , or syntax @xcite , with, recently, the use of pretrained language models @xcite or document-level bias distribution @xcite . bias can be expressed in more subtle ways however. in the excerpts below (figure 1 ), discussing the 2019 virginia beach mass shooting foot_0 , we can clearly identify the difference in coverage, with specific lexical choices ("epidemic", "refuse to cover") but also different ways of presenting the event: the style is either descriptive (bbc) or more emotional (wp, townhall) ; the writer insists on particular topics or angles (use of silencers, weapon prohibition). the choice of topics is indeed an important aspect of information manipulation @xcite , and has also generated nlp work, still lexically focused @xcite @xcite "the virginia beach shooter put a sound suppressor (...) so that the death shots were muffled, perhaps denying others the warning that would have allowed them to escape. it is long past time to remove the silencer that seems to suppress action on gun-control legislation, to treat mass shooting as the epidemic it is, and do everything possible to save lives.'" (washington post, left-leaning) "the attack began shortly after 16:00 (20:00 gmt), at virginia beach municipal center, in an area which is home to a number of city government buildings. the area was put into lockdown by police and employees were evacuated. 'we just heard people yelling and screaming at people to get down,' megan banton, an administrative assistant in the building, told local television news station wavy." (bbc, center) "the chilling fact is that mass public killers are attracted to targets where people can't defend themselves. (...) ninetyeight percent of us mass public shootings since 1950 have occurred in places where people weren't allowed to defend themselves. but the news media refuses to cover this fact, which illustrates the need for self-defense, not for more gun control that doesn't work." (townhall, right-leaning) in contrast, we investigate the task of predicting political orientation of news articles, while trying to consider global argumentative aspects instead of local, lexical ones. this classification task consists in predicting the political leaning of an article by considering, in our case, 3 political classes @xcite . since text-level discourse analysis is still a difficult problem @xcite , our architecture encodes a document while automatically inducing latent structural dependencies as in @xcite , with a focus on elementary discourse units instead of sentences. we hypothesize that structural information can help identify political sides and give insights into aspects related to the argumentative nature of different media. we evaluate our approach on news articles @xcite and also perform a preliminary interpretability study. our contributions are: (i) a model predicting political orientation of news articles, inducing a latent structure over discourse segmented texts, with state-of-the-art results ; (ii) a preliminary analysis of the impact of lexical and structural information for bias detection. our code is available at: https://github.com/neops9/ news_political_bias . ## related work there are multiple ways to consider the task of classifying political ideologies, especially by varying the number and type of classes, and the level of analysis. for example, one semeval 2019 shared task focused on identifying hyperpartisan articles @xcite . political bias can also be characterized by locating "propaganda techniques" in texts, as in the semeval 2020 shared task @xcite . here, we consider the task proposed by @xcite based on 3 classes (left, center, right). a similar task was also considered in @xcite , but their dataset is not available for comparison. in addition, @xcite explore methods that prevent the model from using media-related information while remaining based on other lexical and syntactic ones (see section 3). they report at best 51.41% in accuracy. contrary to previous studies based solely on lexico-syntactic information, we hypothesize that document-level organization is crucial. rather than relying on low-performing discourse parsers, we test liu and lapata (2018)'s approach: structural dependencies over sentences are induced while encoding the document. their results indicate that the learned representations, without ever exposing the model to linguistic annotations or an external parser, achieve competitive performance on a range of tasks while arguably being meaningful. this approach is effective for summarization with the learned structures, while less complex than classical ones, capturing consistent information @xcite @xcite . a similar approach was shown to be effective for detecting fake/real news articles @xcite . while focused on discourse-level phenomena, previous studies use sentences as basic units. we experiment with a fine-grained level, discourse segments, provided by a state-of-the-art segmenter. ## model in @xcite , the sentences in each document are composed of sequences of static word embeddings that are fed to a bi-lstm to obtain hidden representations used to compute the sentence representations, that are then passed through another bi-lstm to compute the document representation. at both levels, representations are built using the structured attention mechanism allowing for learning sentence dependencies, constrained to form a non-projective dependency tree. finally a 2layer perceptron predicts the distribution over class labels. we modify the model to include the improvements proposed by @xcite . in particular: (i) we remove the document-level bi-lstm, (ii) for the pooling operation, we aggregate over units using a weighted sum based on root scores, instead of a max pooling, (iii) we perform several additional levels of percolation to embed information from the children's children of the tree, and not only direct children. on top of that, we skip the sentence-level structure attention as it adds an unnecessary level of composition that was found to have a negative empirical impact on the results. ## dataset and settings allsides dataset the articles are crawled from the allsides website, foot_1 with 192 news sources covering 109 topics. allsides is a platform that offers an analysis of the political leaning of various englishlanguage media at the article level by annotating them with 5 political classes that cover the whole political spectrum from the left to the right. the published version of the dataset foot_2 used in @xcite does not match their paper as it includes resp. 2, 817 and 119 additional articles and media. although it complicates results comparison, we kept the published dataset which is large and seems well designed. 4 this dataset comes with two organizations: article-based or media-based. we chose the latter (30, 246 articles) where media present at training time are excluded from evaluation, which avoids evaluating the model on articles that come from media already seen during training. for complexity reasons, we removed from the training set the longest articles in terms of number and size of segments, using a threshold of 100. the final dataset contains 27, 146 articles, see side prec.% recall% f 1 % left 67.39 27.19 38.75 center 39.59 72.76 51.28 right 66.53 61.74 64.05 2020 ) should be considered with caution. in particular, since they do not yet provide an implementation to replicate their experiments, we cannot control the overspecialization issues. regarding biases towards the topics covered, we rely on the analysis by @xcite for their dataset: they showed that topics covered are fairly represented in each class and thus that it should not significantly impact the model decisions. we also want to give here some insights into the model by an analysis with interpretablity methods at the lexical level but also with respect to the induced structure. saliency map a saliency map in nlp is a method for visualizing a deep learning model by computing relative importance of each token (word) in the input based on gradients @xcite . it allows us to identify the lexical cues that provide partial understanding of the decisions made by the model. here, we considered the vanilla gradient approach @xcite , focusing on the gradient of the loss with respect to each token embedding. from these, we can first clearly assess the positive impact of the aa method. lexical cues used by the model without aa, such as the name of the media source, are no longer as relevant for the prediction, although still present. we notice that the model focuses on specific lexical fields depending on the political orientation of the article, such as health, numbers/statistics, economy, for left, center and right leaning articles respectively. we found that crucial information for the model are the mentions or quotes of political figures (e.g., donald trump, hillary clinton, @realdonaldtrump, barack) by media sources of the same political side, but they also represent an important source of errors when it appears in articles of the opposite side as the model tends to use this information alone without considering its context. it also confirms our intuition that there is relevant information in the middle and at the end of articles, even though the model usually focuses on small portions of text, and it explains why reducing the entry size results in a loss of performance. an example is provided in appendix a as heatmap. structured attention regarding structured attention, we extracted the maximum spanning trees from the attention scores using the chu-liu-edmonds algorithm @xcite . an example of dependency tree is given in appendix a. for a first qualitative analysis, we looked at some statistics following @xcite methodology. in particular, we measure the average height of trees (10.68), the average proportion of leaf nodes (0.77), and the average normalized arc length (0.35). statistics per class are equivalent. the learned trees have complex (non-flat) structures which show that relevant information to the model has been encoded in them in contrast to the results obtained by @xcite . we observed that they have marked differences with "natural" structures, such as distant links and it could be interesting to add more constraints. ## conclusion we proposed an original approach for predicting the political orientation of newspaper articles based on learning a latent structure showing the importance of considering elementary discourse units over sentences to include the argumentative dimension, allowing for large improvements over past approaches. we provide preliminary qualitative results on interpreting the predictions to characterize bias. further work will focus on relying on contextual pretrained models while overcoming limitations on document size, and improving output structures and analyses.