Predicting Sentence-Level Factuality of News and Bias of Media Outlets (2023)

Abstract: Automated news credibility and fact-checking at scale require accurate prediction of news factuality and media bias. This paper introduces a large sentence-level dataset, titled “FactNews”, composed of 6,191 sentences expertly annotated according to factuality and media bias definitions proposed by AllSides. We use FactNews to assess the overall reliability of news sources by formulating two text classification problems for predicting sentence-level factuality of news reporting and bias of media outlets. Our experiments demonstrate that biased sentences present a higher number of words compared to factual sentences, besides having a predominance of emotions. Hence, the fine-grained analysis of subjectivity and impartiality of news articles showed promising results for predicting the reliability of entire media outlets. Finally, due to the severity of fake news and political polarization in Brazil, and the lack of research for Portuguese, both dataset and baseline were proposed for Brazilian Portuguese.

## introduction automated fact-checking and news credibility have become undoubtedly an important research issue mainly due to the potential for misinformation to spread in the modern media ecosystem @xcite . furthermore, although fake news is spreading on social media, it is necessary a source media where they would have been posted originally. since websites have published low-credible news in the past, it is likely to happen again @xcite . nevertheless, automated news credibility to assist human efforts and increase the understanding of the news ecosystem as a whole still requires urgent improvements @xcite . nowadays, fact-checking organizations have provided lists of unreliable news articles and media n. sentence-level news article label title president lowers brazil's image with repeated misinformation and does not receive attention from global leaders. biased s1 president jair bolsonaro touch a sore point of europeans when he pointed out that the increased use of fossil fuels is a serious environmental setback, in his opening speech at the un general assembly, tuesday (20). biased s2 germany received criticism at the un for the investment agreement with senegal for the production of gas in the african country. factual s3 "this constitutes a serious setback for the environment", he said, referring to the europeans quotes s4 however, bolsonaro signed measures contrary to environmental protection during the four years of the brazilian government. factual s5 there is a huge difference between speaking at the un and being heard at the un. ## factnews dataset we collected, annotated, and released a new dataset titled factnews, which consists of a sentence-level annotated dataset in brazilian portuguese that contains 6,191 annotated sentences, as follows: 4,302 sentences annotated as factual spans; 1,389 sentences annotated as quotes, and 558 sentences annotated as biased spans. the entire dataset-building process lasted an average of six months. a dataset overview is shown in ## results and discussion for sentence-level media bias prediction, the best model obtained 67% of f1-score. notably, the partof-speech model presented competitive results for both tasks in contrast to the subjective lexicons, which obtained poor results for both tasks. ## conclusions since low-credibility media outlets may potentially be targeted for the spreading of misinformation, we study the factuality of news reporting and bias of media outlets at the sentence-level for fine-grained source reliability estimation. we further provide a new data resource and baselines for brazilian portuguese low-resourced language. we first created a large and manually-annotated dataset for sentencelevel factuality and media bias prediction. then, we provided a detailed data analysis, demonstrating the reliability of the annotation schema and models. finally, baseline models for sentence-level factuality and media bias prediction by bert were presented in order to provide an accurate score of the reliability of the entire news media. results also showed that biased spans are more numerous in words and emotions compared to factual spans. moreover, media outlets presented different proportions of bias, and its distribution in news articles may vary according to the domain, in contrast to factual spans. we also concluded that expert annotators are more successful to identify media bias.