An Effective Approach for Informational and Lexical Bias Detection (2023)

Abstract: In this paper we present a thorough investigation of automatic bias recognition on BASIL, a dataset of political news which has been annotated with different kinds of biases. We begin by unveiling several inconsistencies in prior work using this dataset, showing that most approaches focus only on certain task formulations while ignoring others, and also failing to report important evaluation details. We provide a comprehensive categorization of these approaches, as well as a more uniform and clear set of evaluation metrics. We argue about the importance of the missing formulations and also propose the novel task of simultaneously detecting different kinds of biases in news. In our work, we tackle bias on six different BASIL classification tasks in a unified manner. Eventually, we introduce a simple yet effective approach based on data augmentation and preprocessing which is generic and works very well across models and task formulations, allowing us to obtain state-of-the-art results. We also perform ablation studies on some tasks to quantify the strength of data augmentation and preprocessing, and find that they correlate positively on all bias tasks.

## introduction news outlets have expanded to become one of the most influential and prominent platforms within mass media. news articles play a central role in transforming individual and public opinion @xcite . public opinion of journalists influence viewers to become biased towards a particular issue. the consequence of media bias is massive, and raises questions about the credibility of news. for example, @xcite reported that media bias led to the election of the wrong candidate, and according to @xcite , voters are always well-informed with unbiased rather than biased media outlets. moreover, biased media outlets have also been found to provide less information, which reduces voter's welfare @xcite . misinformation has enormous potential in changing the individual and public beliefs, expectations, or desired conclusions. the harmful biases contained in news media require rigorous analysis to be detected and quantified, but once addressing these issues can improve the quality of research process to maximize the accuracy and credibility of research results @xcite . to enhance transparency and reliability in promoting accurate information, it is important to realize bias in machine learning methods and how humans perceive bias @xcite . seminal work by @xcite has been a key contribution in bias detection in news, with the introduction of the basil dataset. critically, to the best of our knowledge, this dataset is the first to be annotated with different kinds of bias. this is due to the fact that, as pointed out by @xcite , some kinds of bias can only be analyzed in a broader context, because a given sentence becomes ambiguous in isolation at any given time. in this sense, their proposal was to differentiate informational bias from lexical bias. on one hand, the former is usually presented using speculative, imperative, and tangential clauses that convey information in a factual and neutral way (van den @xcite to sway readers' opinions towards news entities @xcite , therefore depending mostly on the context. on the other hand, lexical bias instead depends on linguistic attributes like word choice and syntax and originates from content realization or how things or events are expressed, and is generally not depending on the context @xcite @xcite @xcite . ## related work there has been a growing interest in the investigation of linguistic information presented by neural models @xcite . a study by jia et al. (2019) worked on the bias and distortion of online commentary information based on online reviews, and illustrate online review components significance over increased reputation and false reviews. hovy and prabhumoye (2021) study five sources of bias including data, annotation process, input representations, models, and research design where research design is the most difficult to detect because it requires systematic analysis and subjectivity of human perceptions. the work on basil dataset in the literature is not exhaustive to the best of our knowledge. basil dataset provided by fan et al. (2019) use bert and roberta for informational and lexical bias detection while treating sentences in isolation, whereas informational bias is also explored with different types of contexts like textual, article, event and domain using bilstm's, bert and event context-inclusive model (evcim) inspired by context aware model of papalampidi et al. ( 2019 ), we will omit a detailed explanation of these models, referring readers to (van den berg and markert, 2020). @xcite demonstrate that informational bias is more challenging to detect due to its dependence on content selection as compared to lexical bias. another study by @xcite introduce a general purpose misinformation unifiedm2 model for bias detection in basil and handle tasks like fake news, clickbait and rumors. contrastive learning and graph attention network in the mul-tictx (multi-level context) model uses triplets of basil to detect informational bias as proposed by @xcite . similarly, bias sentence identification is also studied by @xcite through local and global discourse structures using roberta for addressing bias. we compare our results with the aforementioned current state-ofthe-art methods to investigate the significance of our proposed approach. another topic relevant to our work is data augmentation. in general, data augmentation in nlp is at an emerging stage compared to computer vision @xcite . among data augmentation techniques in nlp one that has recently shown excellent results in the context of machine translation is backtranslation @xcite . the idea is to pair monolingual data with an automatic backtranslation generated by a given model or external system, which led to considerable performance improvements on neural machine translation, showing that even for small amounts of in domain monolingual data, backtranslation is effective for domain adaptation. more recently, @xcite scale backtranslation to hundreds of millions of monolingual sentences and achieve substantial improvements. the same approach is more recently been successfully adapted to other task. for example, @xcite focus on sentiment classification and use backtranslation on only 500 examples, obtaining an 8.1% improvement over the augmentation-free baseline based on ulmfit @xcite on the imdb dataset. our work is similar to this, but we adapt the approach for bias detection. finally, our work is related to stemming, the process of linguistic normalization where variant forms of a word are converted to a stem or root word. there are different methods for stemming, but most of them follow a rule-based or linguistic approach, each one with its own advantages and limitations @xcite . relevant for us is the fact that stemming has been shown to add semantic value in feature selection, as for example @xcite proved that stemming of composite words greatly improves classification of fake news. moreover, @xcite showed that cleaning and stemming resulted in the greatest model performance on the medical domain for the task in mortality prediction on icu (intensive care unit) patients. we refer readers to a thorough survey of stemmers spanning over the past 50 years by @xcite . ## proposed approach the bias annotation spans on the informational level (basil) dataset, provided by @xcite , is based on three news sources, i.e., huff-ington post (hpo), fox news (fox), and new york times (nyt) from 2010 to 2019, containing 300 news articles with 100 triplets of news articles taken from each news source. basil contains both binary sentence classification labels of informational and lexical bias together with span level annotations for token classification. an isolated sentence is biased if it contains at least one bias span. there are 7,977 sentences in basil having 1,249 sentences with informational and 478 sentences with lexical bias @xcite . basil has more prevalence of informational bias than lexical bias. it covers news articles of reasonable time i.e., 10 years, representing conservative from fox, neutral from nyt, and liberal from hpo, respectively. since informational bias is presented in a factual and neutral way, there is increasing prevalence of informational bias in news media, as evidenced by @xcite . this factual reporting of informational bias makes its prediction more challenging on a sentence level. on the other hand, lexical bias is reportedly easier to capture because of its non-contextual nature, depending mostly on word choice @xcite . for example, a study on tagalog-speaking filipino pre-school children by @xcite showed that lexical bias is related to lexical inventories like nouns, verbs and adjectives produced by young children, with nouns as the leading or dominant lexical bias among children in their narrative production. despite these facts, the detection of both types of bias remains a difficult and challenging task, as evidenced by the study on basil performed by @xcite , which includes approaches for the automatic detection of both types of bias (although not distinguishing them). basil has 7,977 sentences, from which 1,249 and 478 are labeled as containing with informational bias and lexical bias, respectively. following the formulation of @xcite , this leads to 6,250 sentences with no bias. after careful consideration of models proposed in prior work utilizing basil, we note inconsistencies in the task formulation across papers, which are derived from the way in which these labels are interpreted and used. in order to shed light into this issue, we begin by organizing prior work and defining the notation we will utilize in the rest of the manuscript, for the sake of simplicity. we refer to sentences annotated with no bias as 'neu', sentences annotated with we can see that there is a clear disparity in the way in which the bias detection tasks are approached. we further observe that detection of informational bias strikes as being the main focus in literature so far. we surmise that this issue may be due to the fact that lexical bias samples on basil appear to be considerably fewer than informational bias. however, we think that detection of lexical bias @xcite ) should be at least as important as informational bias. while previous results focus mainly on the inf-related settings, we propose to experiment on classifying lex bias from inf as, depicted in the first row of in light of these observations, our work shows the results of several experiments we perform to compare and distinguish different bias in basil, using various settings. to further study the relationship between labels, we also propose a novel three-way classification approach to directly differ-entiate informational, lexical and neutral samples (inf/ lex/ neu), respectively. as a result of our extensive study, we note that in many cases, performance gaps are due to lack of annotated examples for a given label, with this being particularly the case for lexical bias. in light of this issue, in this paper, we also adapt backtranslation as a mechanism for selective data augmentation in bias detection and use google translate python api for backtranslation. our approach is inspired by previous work of @xcite , who proposed a chinese text data augmentation based on back-translation to generate a corpus to enrich the lexical features of text data, and reported increased performance on text classification tasks, especially when training on smaller datasets. back-translation is also a commonplace to improve the performance on machine translation tasks @xcite . our idea is also motivated by seminal work by @xcite who achieved 90% precision for translation of words between english and spanish and found word vectors of both languages contain similar geometric arrangements. we used google translate python api which is accessible free under the mit license. finally, we note that data preparation plays a significant role in machine learning, especially for natural language processing tasks @xcite . inspired by the work by @xcite and ladani and desai (2020), we preprocess sentences in the basil dataset to remove noise. our proposed text preprocessing strategy involves two steps, first the removal of special characters and stop words i.e., words like 'after', 'before', 'the', 'some', 'all', 'such', and second reduction of words into their grammatical root or stem. in the context of bias classification task, as well shown in section 5 we found that reducing the number of dimensions in terms of space (rakholia and saini, 2016) by removing most common words or words that normally carry no meaning has significant impact on bias detection. ## experimental framework in this section, we discuss the experimental framework including the setup, baselines, implementation details and give details of the models we train. ## results & discussion we begin by testing the impact of our proposed data augmentation and preprocessing approaches by running ablation studies on three task settings. we first consider the newly-introduced tasks of inf/ lex and inf/ lex/ neu, in addition to the more standard inf/ oth setting, which is generic and of particular interest in the prior work. we measure the impact of each component of our proposed work by repeating the experiments and dividing our ablation test into two steps. to study and measure the significance of our proposed components, we first experiment only with data augmentation (denoted as 'aug.') which involves the backtranslation of samples with bias, which is followed by our preprocessing step (denoted as 'preproc.') which includes data cleaning, removal of stop words and stemming. the evaluation criteria we used is accuracy as 'acc' and f1-micro score separately on each class of the bias i.e, inf, lex, neu, and oth as explained earlier. from our literature review, as discussed in section 3 we found that many potentially relevant task formulations were missing. we surmise this could be due to the scarcity problem of lexical bias samples in basil, with interest focusing on only informational bias. we therefore propose to tackle all relevant task formulations. concretely, as to the best of our knowledge there is no prior work done to classify lexical bias from informational, we believe we are the first to identify and report experiments on the tasks of inf/ lex as binary classification, as well as inf/ lex/ neu, use a multi-class classification. we further note that our three-way classification task inf/ lex/ neu follows a similar trend, with significant performance improvements due to our data augmentation technique. a sharp rise in accuracy is observed by inclusion of both augmentation and preprocessing, leading to an accuracy of 81.54% when using bert. from prior work on basil as discussed previously and as shown in having established the effectiveness of our proposed techniques, we now move on to compare our models with previous work, as shown in the first task shown in in our second binary task, bias/ neu, we use the same model configurations as @xcite with a batch size of 32, a learning rate of 5 × 10 -6 with 15 as a maximum epoch count. here bias corresponds to the combination of informational and lexical bias sentences, while neu symbolizes only neutral sentences, which therefore corresponds to detecting bias v/s no bias sentences irrespective of their type. in this task, the highest bias f1-score is reported by unifiedm2, and though we see that our proposed bert-based approach does not outperform this model, our results are competitive as we observe we attain a similar performance. furthermore, we note that our results in this task indicate that small focused data augmentation techniques as ours could be nearly as effective as more complicated training procedures including many tasks, as the approach proposed by unifiedm2. to detect lexical bias alone from neutral sentences, we study the task of lex/ neu. as shown in finally, we consider the task of inf/ oth, where we find most of previous work has focused. concretely, for this task, we compare our bertbased model with six previous models from the literature. as can be seen, we find that our model is able to outperform all existing prior work, achieving the best performance in terms of with inf f1-score of 50.9. our model is followed by the roberta model fine-tuned by (van den berg and markert, 2020) with inf f1-score of 49.89, where roberta by @xcite as third with 46.47 and multictx follows in the fourth position with f1-score of only 46.08. we believe this indicates that our approach is also better at recognizing informational bias as a type of misinformation. furthermore, we note that for the inf/ oth task, many of our considered baselines are based on pre-trained models similar to ours, such as bert or roberta. as our model is based fundamentally on the same deep learning model, we believe these results suggest that our augmentation and pre-processing approaches might work for those models also. this adds to our observations derived from ## conclusion this paper presents different techniques of phrasing bias to tackle media bias in new outlets. we propose an approach that relies on current neural network models to capture sentence level biased language. we defined how data augmentation is applicable to less frequent bias in news articles and measure the effect of its performance across different models. human annotation is costly and conditions where obtaining new misinformation samples is difficult, our approach is significant to such real life cases. since our proposed approach involves simple feature extraction techniques to tackle a particularly small and unbalanced biased dataset, we believe our work can be used to mitigate bias and improve the quality of the model's predictions in real-world scenarios. we identify some novel tasks in basil and our augmentation technique effectively detect informational and lexical bias sentences simultaneously, while also outperforming in other tasks. in our work, we incorporate different methods to process bias and illustrate the importance of our proposed components. a key distinguishing feature of our work is the removal of inconsistencies of prior work in reporting and evaluating bias types of basil. ablation studies are also performed by varying training data in different tasks and our technique suggest significance of each proposed component in different experimental settings. we found the performance improvement of our proposed approach in almost all tasks as compared to several state-of-the-art techniques, hence this proves that our methodological standpoint in using small augmented data is well-aligned in finding informational and lexical bias sentences in different classification tasks. similarly, our work tries to propose a way of regulating different task formulations of basil which are unclear in prior work. we intend to explore context in basil news articles as future work, besides trying other feature selection techniques. we believe further parameter optimization and fine-tuning for different task formulations can also improve the results.