C zech-ing the News: Article Trustworthiness Dataset for C zech (2023)

Abstract: We present the Verifee dataset: a multimodal dataset of news articles with fine-grained trustworthiness annotations. We bring a diverse set of researchers from social, media, and computer sciences aboard to study this interdisciplinary problem holistically and develop a detailed methodology that assesses the texts through the lens of editorial transparency, journalist conventions, and objective reporting while penalizing manipulative techniques. We collect over 10,000 annotated articles from 60 Czech online news sources. Each item is categorized into one of the 4 proposed classes on the credibility spectrum – ranging from entirely trustworthy articles to deceptive ones – and annotated of its manipulative attributes. We fine-tune prominent sequence-to-sequence language models for the trustworthiness classification task on our dataset and report the best F-1 score of 0.53. We open-source the dataset, annotation methodology, and annotators’ instructions in full length at https://www.verifee.ai/research/ to enable easy build-up work.

## introduction donald trump has called journalists and news outlets "fake news" nearly 2, 000 times since the beginning of his presidency, averaging more than one daily broadside against the press between 2016 and 2020 @xcite . because of trump, the term fake news underwent a fundamental change in its meaning. at first, it referred to a satirical and ironic genre of fictional news designed to entertain the audience. the original "fake news" have appeared on tv shows such as saturday night live on nbc or in print, such as the onion. however, during trump's campaign for the us presidential election in 2016 and his presidency, the concept of fake news became an integral part of his political communication. it aimed to discredit critical journalistic content or the whole news media as "fake media." the successful stigmatization strategy of "fake news" has led to a fascination with this phenomenon in the public discourse and science. fake news has become a label for false news and a synonym for both disinformation and misinformation. this has strengthened the binary perception of the credibility of information in a true-false dichotomous perspective. however, this reductionist approach has become a barrier to understanding the more profound meaning that the buzzword "fake news" covers. if we want to examine the credibility of the news content seriously, it is not possible to adopt the binary approach of either truth or lie. by creating the verifee dataset, we try to overcome the interdisciplinary barrier between social sciences (especially journalism and media studies) and computer science. this barrier prevents specialists in automated or robotic journalism from adopting a more analytical approach to various types of information disorders that we have become used to labelling with the general term "fake news". ## related work herein, we first review the current literature focusing on disinformation and misinformation in the journalistic ambit. we later provide an overview of existing methods treating these phenomena within the artificial intelligence (ai) and natural language processing (nlp) research communities. we first list some of the already available datasets and then focus on the architectures solving the tasks of fake news detection and automatic fact-checking. the task of fake news detection resides in classifying whether a given news article (or another medium, such as a tweet) is considered fake (disinformative) or truthful (credible). there is no consensus in the literature on what specific parameters constitute this distinction, but truthfulness is usually considered the primary one. some approaches recognize more fine-grained scales with specific classes (e.g., tabloid news, mixed reliability news), whereas others are binary (including fake and credible classes). either way, the sole text is considered. meanwhile, the task of automatic fact-checking requires a source of truth to which the news article is compared. the task then lies in determining whether the article is supported by facts therein. hence, one can consider this task a specific variant of stance detection focusing on news media and large-scale ground-truth databases. we review datasets and approaches in both of these tasks, as our dataset lies somewhere in between. ## trustworthiness assessment methodology having familiarized ourselves with the current state of research, we concluded that the best way forward is to build upon the previous work and introduce a new language-agnostic methodology for classifying news articles. the primary motivation for this was the inability of prior approaches to fully reflect the complexity of the problem in terms of media studies and fully appreciate each article uniquely and independently of its source. we hope to provide better data for ai-based tools concerned with handling dubious news articles with this methodology. below, we introduce the basic framework of our methodology. its complete overview is available in appendix a. ## dataset we collected a dataset of 10, 197 czech news articles. each entry in the dataset consists of the article's text, html source, title, description, authors, source name, url, main image, and the hillary clintonová je nebezpečím pro svět, udeřila ve francii le penová vůdkyně francouzské národní fronty marine le penová poskytla rozhovor francouzské verzi proruského zpravodajského russia today. konstatovala v něm, že by se jako příští prezidentka francie mohla přimluvit za uznání připojení ukrajinského poloostrova krym k rusku. za velký problém pro svět by prý považovala… hillary clinton is a danger to the world, le pen slams in france french national front leader marine le pen gave an interview to the french version of the pro-russian outlet russia today. in it, she stated that as the next president of france, she could argue for the recognition of the annexation of the ukrainian peninsula of crimea to russia. she said that she would consider the election of… original czech data translated into english overall class: misleading source: parlamentní listy tags: 􀯩 missing author 􀇾 clickbait 􀯐 logical fallacies (+ image & annotator info) annotations mělo by se ve školách s psaním a čtením učit i programování? kovy se ptal ženy, která mění česko tvrdí, že it není jenom pro kluky. dita přikrylová je datová analytička a zakladatelka hnutí czechitas, které se snaží bourat předsudky a otevírat ženám svět, jenž bychom v rámci našich předsudků přiřadili spíše klukům. do dalšího dílu pořadu v centru si ji pozval karel "kovy" kovář… should we teach coding at schools along with reading and writing? kovy asks the woman who is changing czechia she says that it is not just for boys. dita přikrylová is a data analyst and founder of the czechitas movement, which seeks to break down prejudices and open up a world for women that we would rather assign to boys. karel "kovy" kovář has invited her to the next episode … original czech data translated into english overall class: trustworthy source: seznam zprávy tags: 􀯩 missing author (+ image & annotator info) ## experimental results we conduct baseline results by fine-tuning generalpurpose architectures. ## ethical discussion and limitations due to the high-impact nature of the solved task, we review the ethical considerations made during this research project. additionally, we outline further steps to ensure safety and transparency beyond publication, as well as recommendations for buildup work. first, let us focus on the presence of biases in the data. we put extensive procedures in place even at the very start of the project. by inviting media researchers into our core team, we wanted to minimize misunderstandings and mistakes that scientists from the field of computational linguistics could easily make when assembling the methodology for the task of trustworthiness assessment due to their limited knowledge of the current literature and theory in the area of journalism. prior to the data annotation, we invited scholars in media studies and journalists from the industry to a series of workshops, where we asked them to submit feedback and discuss the methodology. based on the assembled comments, we kept updating it until a general consensus was reached. in terms of the annotation process itself, multiple safeguards have been employed to prevent annotators' bias towards specific sources or authors (that may affect the classification). second, let us shift towards the ethics of using any technology built around this data in the wild. we want to stress that anyone using this dataset for the purposes of creating a trustworthiness classification system should provide transparent information to the users that this process is automatic and hence faulty to a certain extent. we must note that it still needs to be determined how models trained on this data generalize for future articles (i.e., news about topics and events they have not encountered in the training set) and news sources not included in the training set. a study into these should be conducted prior to making this technology available unrestrictedly to the public. despite bearing these safety questions in mind is crucial, such systems can eventually be great assistive tools for people reading news stories online. the potential benefits of such technology should support initiatives to safeguard it first and establish public and academic trust. ## conclusion this work presents a novel methodology for classifying news article trustworthiness and presents a multimodal dataset of 10, 197 czech news articles with respective annotations. unlike previous methods that classify all texts from a given media outlet with the same class, we treat the articles on an individual level. the high inter-annotator agreement shows that our methodology constitutes a good feature-based framework, leaving little to no room for personal annotators' inducement. to the best of our knowledge, we are the first to include media and computer science researchers in the core team when developing a similar dataset. additionally, all of our annotators were journalism students. as our methodology underwent extensive feedback loops with professionals in the industry, we hope to establish a new interdisciplinary standard for future related works to follow. we provide benchmark results on our dataset using 5 different classifier architectures and obtain promising results -the best-performing roberta model achieves a testing f-1 score of 0.53. we open-source the complete dataset and encourage researchers to undertake similar initiatives in new languages and social contexts, especially lowresourced ones. since the framework derives all parameters based on the text contents, it is languageagnostic. hence, minimal additional methodological work is necessary before new annotations. in future work, we intend to study the generalization abilities of systems trained using this data and the application of task-specific architectures. moreover, we wish to further explore the potential of multimodality that our dataset offers and analyze the attached images.