An Integrated Approach for Political Bias Prediction and Explanation Based on Discursive Structure (2023)

Abstract: One crucial aspect of democracy is fair information sharing. While it is hard to prevent biases in news, they should be identified for better transparency. We propose an approach to automatically characterize biases that takes into account structural differences and that is efficient for long texts. This yields new ways to provide explanations for a textual classifier, going beyond mere lexical cues. We show that: (i) the use of discourse-based structure-aware document representations compare well to local, computationally heavy, or domain-specific models on classification tasks that deal with textual bias (ii) our approach based on different levels of granularity allows for the generation of better explanations of model decisions, both at the lexical and structural level, while addressing the challenge posed by long texts.

## introduction in an expanding information-based society, where public opinion is influenced by a plurality of sources and discourses, there is growing concern about fair information sharing. biased speech, slanted presentation of events are inevitable, whether intentional or not, but must be transparent to ensure a more democratic public space. this has motivated substantial work on text classification to identify political orientation, what stances are supported by a text, or to characterize misleading or fake information @xcite . it is also important that such methods can provide justifications to their decisions, both to understand what linguistic expressions are characteristic of certain positions, and also to provide some transparency in the analysis itself. explainability of supervised models is now a large subfield addressing this concern, with methods providing justifications, mostly in the form of relevant tokens in the case of textual tasks, e.g. @xcite . in this work, we contribute to both these lines of research by proposing an integrated approach for predicting and explaining political biases, where the structure of the document can inform the proposed bias characterization, as opposed to current approaches only relying on lexical, local cues. indeed, by focusing on local formulation, existing research @xcite ignores that political expression also relies on argumentation, i.e. the way information is presented. example 1 is segmented into elementary discourse units (edus), the minimal spans of text to be linked by discourse relations as described e.g. in the rhetorical structure theory @xcite . the discourse structure built upon these segments represents how information is conveyed in a right-leaning text about climate and can inform on how the information is presented (why the climate is not a problem, what opposing argument the writer wants to highlight), and also to detect the most important spans of texts. example 1. [there's nothing abnormal about the weather this january,] 1 [it's just part of the earth's natural climate patterns.] 2 [the mainstream media is just pushing the idea of climate change] 3 [to push their own agenda.] 4 to the best of our knowledge, we are the first to investigate discourse-based information for bias characterization, and we do so through: (i) a segmentation of the texts based on discourse units rather than sentences, (ii) experiments on discourse connectives that can be seen as shallow markers of the structure, (iii) and crucially, a model based on latent structures, as a proxy for discourse structures, that can help the prediction and provide a different sort of input for explainability methods. furthermore, while recent progress on text classification has been largely due to the wide-spread use of pretrained language models, fine-tuned on specific tasks, they remain limited in terms of input size (i.e. 512 sub-tokens in general) and can-not easily deal with phenomena that relate elements far apart. long texts are also problematic for many explanation methods. our proposed approach addresses this limitation on both sides. the code is available at: https://github.com/neops9/ news_political_bias . ## related work the prediction of the political orientation in texts has long been of interest in political science @xcite , and has generated growing interest in nlp, either for classification at document level, e.g. detecting extreme standpoints @xcite or more general left/center/right orientation in news @xcite @xcite , but also at a finer-grain local level, locating specific framing @xcite , or various linguistic devices such as "propaganda techniques", as in the semeval 2020 task @xcite . for a more general view, see the survey in @xcite . recently, @xcite have developed a language model over roberta @xcite , fine-tuned on a large corpus of news to address both stance and ideology prediction, by incorporating new "ideology-driven" pre-training objectives, with very good results. in contrast, we develop a generic approach that could be applied as is to new classification tasks. aside from approaches whose objective is just prediction of an orientation, some studies aim at characterizing bias, and rely on lexical statistics or surface cues @xcite . in contrast, we want to investigate other factors as well, at a more structural level, mainly document-level organization aka discourse structure. automated discourse analysis is the subject of a rich body of work but current parsers still have rather low performance and weak generalization. this is why we took inspiration from @xcite , who use structural dependencies over sentences that are induced while encod-ing the document to feed downstream supervised models. their results indicate that the learned representations achieve competitive performance on a range of tasks while arguably being meaningful. this approach is effective for summarization with the learned structures, while less complex than relying on rhetorical relations, capturing consistent information @xcite @xcite . similar results were found for fake news classification @xcite . our model relies on these approaches, but adds a finer-grain level of analysis relying on elementary discourse units. the last aspect of our approach is the use of explainable methods to characterize bias. we propose an integrated approach where a classification model is used with methods to explain its decision, thus providing cues about the way bias is present and detected in texts. numerous explainability methods have been proposed in recent years, most of which are amenable to being used on text classification tasks. almost all of them are local i.e. provide information about the role of separate parts of the input for a given instance only, e.g. input tokens most relevant to a model's prediction for textual tasks. these methods can be either black box methods, operating only on predictions of the models @xcite , or can observe the impact of the input on some of its internal parameters @xcite . we extend the use of such methods to take into account structural elements. although some studies have recently investigated how structural / discourse information is encoded in pretrained languages models @xcite , to the best of our knowledge, we are the first to explore textual explainable methods not relying only on surface form information. this is crucial for long texts, as methods such as lime @xcite that rely on sampling word perturbations can become expensive for high token counts. ## integrated bias detection and characterization our approach is based on a model that predicts a bias while inducing a structure over documents, and explanation methods that could either take as inputs simply the tokens, the edus, the sentences, or that could be based on the induced structures, see figure 1 . in this section, we describe our model for predicting bias, on which we rely to produce structure-based explanations. ## lexical and structural perturbation-based explanations among the numerous existing methods for interpreting a model's decision, we chose to focus on so-called black box approaches, only relying on a model output predictions, and not its internal representations, for more generality. however, the most popular black box approaches, lime @xcite , anchor @xcite or shap (lundberg and lee, 2017) rely on lexical features when applied to textual tasks, looking for relevant subsets of features or using perturbations by removing/switching words in the input which makes them computationally expensive for high token counts, or forces approximation via sampling, which still has to be representative enough to be useful. of these methods we chose to only consider lime, which is intrinsically based on sampling and has been shown by @xcite to have the best or near-best performance on their metrics, and thus present a good compromise. lime works by learning a simple model around an instance, which approximates the prediction of the model in the "neighborhood" of the instance. the neighborhood of an instance is sampled by slightly perturbing the input with respect to some features, words in the case of textual models, yielding a set of (perturbed) instances. then a simple linear model is fitted on these instances to match the model predictions, with a weight given to the instances according to their distance from the original instance. the parameters of the simple model then yield importance scores for the input features, and the best ones are chosen as an "explanation" of the decision on the original instance. despite its usefulness, lime has some known limitations, regarding the cost of the sampling process (molnar, 2022, section 9.2.5) or the robustness of the explanations @xcite . the main issue is that the quality of the explanations highly depends on the amount of generated perturbed samples, to be representative of the model's behavior, and to avoid spurious or not robust explanations. for texts, where features are words, this can mean a high computational cost, especially for long documents, since the number of possible perturbations of a text grows exponentially with its size. we thus propose four strategies to reduce this cost and still produce relevant explanations, by focusing on different levels of granularity. ## explanation evaluation metrics evaluating the explanations is an important challenge, and common practices mostly depend on costly human judgments. here we rely on the diagnostic properties proposed by @xcite in the context of text classification. we discarded two measures that cannot be computed: the agreement with human rationales measure, since we do not have access to human annotations for the explanation of political datasets, and the rationale consistency measure, since it is meant to compare an explanation method across different models. we consider that a document is composed of a set of features, and that our explanation method generates a saliency score for each of them. #bert tokens #edus #sent. allsides 1257 ± 863 58 ± 44 32 ± 25 c-politics 1008 ± 1106 100 ± 112 20 ± 24 hp 780 ± 691 81 ± 74 25 ± 24 confidence indication (ci) when generating an explanation, the feature scores for each possible class can be computed. it is then expected that the feature scores for the predicted class will be significantly higher than those of the other classes. if not, this should indicate that the model is not highly confident in its prediction, and the probability of the predicted class should be low. we can then measure a confidence indication score as the predictive power of the explanation for the confidence of the model. predicted confidence is computed from the distance between saliency scores of the different classes and then compared to actual confidence by using the mean absolute error (mae). faithfulness faithfulness is an indication that features selected in an explanation were actually useful for the model to make a prediction. it is measured by the drop in the model's performance when a percentage of the most salient features in the explanation are masked. starting from 0%, 10%, up to 100%, we obtain the performance of the model for different thresholds. from these scores, the faithfulness is then measured by computing the area under the threshold-performance curve (auc-tp). dataset consistency (dc) dc measures if an explanation is consistent across instances of a dataset. two instances similar in their features should receive similar explanations. similarity between instances is obtained by comparing their activation maps, and similarity between explanations is the difference between their saliency scores. the consistency score is then the spearman's correlation ρ between the two similarity scores. the overall dataset consistency is the average obtained for all the sampled instance pairs. ## datasets we evaluate the effectiveness of our approaches on three english-language datasets foot_3 which contain annotations of political leaning (bias) of long news articles, and thus particularly relevant to the context of this study. lengths of documents are shown in allsides this media-based news articles dataset proposed by @xcite foot_4 contains 30, 246 articles with 3-class annotations: left, center, right. media present at training time are excluded from evaluation. the articles were crawled from allsides foot_5 which is a platform that offers an analysis of the political leanings of various english-language media at the article level. an article is labeled by the political positioning of its media. hyperpartisan (hp) a binary classification task @xcite of predicting whether a given news article is hyperpartisan or not (takes an extreme left-wing or right-wing standpoint), task 4 of semeval-2019. we considered the dataset containing 1, 273 manually annotated articles. ## experimental settings baselines for allsides and hyperpartisan, we compare to the results obtained by the authors of the datasets, and the winners of the task (hp). we also compare to three additional transformerbased baselines on the three tasks, for which we fine-tuned a classification model (on a single run): (1) roberta-base @xcite (2) longformer-4096 @xcite , a language model designed to handle very long sequences of text, up to 4096 tokens (3) politics @xcite , a state-of-the-art language model built over roberta-base for political ideology prediction, pretrained on more than 3.6m news articles (see above). roberta and politics are fine-tuned on the whole input using a sliding window of size 512 and an overlap of size 64; we built on @xcite 's implementation foot_7 . all baselines and proposed models have similar numbers of parameters (cf. the appendix). for the explanations, we compare to the original version of lime for text classification, which is based on words perturbation, and a random explanation on the whole input. settings for the classification model, we built on @xcite 's implementation, foot_8 itself based on liu and lapata (2018)'s. we adapted the code according to the modifications and additions proposed in our approach, as detailed in section 3.1. hyperparameters were set using grid search and are the same for all tasks ( evaluation we evaluate two versions of the classification model: segmentation into sentences, or into edus (on a single run). we report accuracy as it is the standard measure in previous work on these tasks. we built on the lime python package foot_9 to implement our methods (section 4). we generate and evaluate explanations on 100 documents from the test set for 1, 000 and 10, 000 perturbed samples and compute a score for each feature. explanations are generated for our trained classification model with edu segmentation (section 3.1). the confidence interval for the evaluation of the explanations is only given for the baseline (lime words) for 10 generations. since each of the proposed improvements has a reduced perturbation space relative to the baseline, which is the impact factor of the variance, and to avoid a disproportionate computational cost, we consider that the confidence interval will be at worst equal or better, and therefore we do not give it for all experiments. ## results results obtained for the different classification tasks are given in regarding our structured approaches, we can note that despite lower scores compared to pol-itics and longformer, the edu-based version performs better than roberta on corpora with the longest text lengths (i.e. allsides +1.76 points, c-politics +4.37 points). the segmentation into edus significantly improves the results on all tasks compared to the segmentation into sentences (+4.59 points on average), showing the importance of the fine-grain discourse approach. putting these results in perspective, our approach is more generic than politics, as it does not require heavy and domain-specific pre-training, and much lighter than longformer (w.r.t. computational cost). as we are reducing the sampling space in our approaches, we also made comparisons on the number of samples used to generate the explanation for these metrics, between 1, 000 and 10, 000 samples. we notice that the scores obtained by most of our approaches on 1, 000 samples remain better than those of the baseline for 10, 000 samples. this shows that it is possible to generate good explanations, and often of better quality, with a number of samples 10 times smaller, which is a major improvement over the computational cost. ## analysis of explanations by looking at the explanations generated for the different levels of granularity and properties targeted, we can gain some insights about the model's decisions. an important property that must be fulfilled by the explanation is its comprehensibility by a human in order to characterize biases. we propose a qualitative analysis of the explanations and a comparison of the various approaches, both at the lexical and structural level. for the analysis of the structure and its explanation, we compare various statistics following @xcite . average height of trees (6.36), average proportion of leaf nodes (0.87) and the average normalized arc length (0.35) are equivalent between classes, although the right-wing class have slightly more shallow trees. regarding the explanations, the most impactful relationships are mainly located in the first levels of the tree, close to the root, independently of the class. although the explanation by perturbing the tree relations is not the most intuitive at first sight, it allows for a new level of abstraction by providing an understanding of the model's decisions with respect to the induced structure, which combined with other methods of analysis, can reveal additional biases. ## conclusion we propose an integrated approach to both predict and analyze political bias in news articles, taking into account discourse elements. we show that structured attention over edus yields significant improvement at different levels over existing approaches, or comparable results, if lower, with respect to data-or computation-hungrier models. we also proposed new variants for perturbation-based explanation methods when dealing with long texts, both at the lexical and structural level, that would not be possible with the other models. we demonstrate the effectiveness of our system by evaluating it on a series of diagnostic properties, and propose a qualitative analysis and comparison of the various approaches for the characterization of political bias.