IRMA : the 335-million-word I talian co R pus for studying M isinform A tion (2023)

Abstract: The dissemination of false information on the internet has received considerable attention over the last decade. Misinformation often spreads faster than mainstream news, thus making manual fact checking inefficient or, at best, labor-intensive. Therefore, there is an increasing need to develop methods for automatic detection of misinformation. Although resources for creating such methods are available in English, other languages are often under-represented in this effort. With this contribution, we present IRMA, a corpus containing over 600,000 Italian news articles (335+ million tokens) collected from 56 websites classified as ‘untrustworthy’ by professional fact-checkers. The corpus is freely available and comprises a rich set of text- and website-level data, representing a turnkey resource to test hypotheses and develop automatic detection algorithms. It contains texts, titles, and dates (from 2004 to 2022), along with three types of semantic measures (i.e., keywords, topics at three different resolutions, and LIWC lexical features). IRMA also includes domain-specific information such as source type (e.g., political, health, conspiracy, etc.), credibility, and higher-level metadata, including several metrics of website incoming traffic that allow to investigate user online behavior. IRMA constitutes the largest corpus of misinformation available today in Italian, making it a valid tool for advancing quantitative research on untrustworthy news detection and ultimately helping limit the spread of misinformation.

## introduction over the last decade, there has been an increase in worry about misinformation, which has led to numerous studies (e.g., @xcite @xcite . this level of focus is justified by the threat that misinformation poses to individuals, institutions, and society in an increasingly digitalized world. helped by social media capillarity and a lack of gatekeeping, misinformation is eroding long-standing institutional barriers, compromising democratic processes, as happened during the last us presidential elections @xcite , and producing serious sociopolitical uncertainty, as in the examples of global warming and covid-19 vaccines @xcite . currently, there are two main approaches used to detect misinformation online: manual and automatic. the first relies on human effort, mostly represented by fact-checking services that employ experts to manually verify the accuracy of claims, articles, and entire websites. the second is based on the identification of particular textual content features, usually performed through natural language processing (nlp) tools, e.g., deep learning models. because misinformation spreads alarmingly faster than reliable news @xcite , automatic tools allow to detect and limit the spread of false news quickly and without involving costly human effort. these tools are usually trained on large sets of textual data, which are for the most part in english language (see e.g., @xcite @xcite . in a worldwide effort to fight misinformation, resources have been made available for arabic, spanish, portuguese, and german @xcite @xcite . however, to our knowledge, the italian language has been overlooked. several attempts have been undertaken to under-stand misinformation in the italian context (see e.g., @xcite , but these works focus on social media, and the data is not made publicly available. the availability of an open-access dataset would substantially encourage research into the role of misinformation in the italian context. a recent study conducted in italy showed how the inability to recognize false information can obstruct public health campaigns @xcite . misinformation in italy has also been linked to political parties that have governed in recent years @xcite , as well as their voters @xcite . for example, @xcite showed that right-wing parties were responsible for 96% of covid-19-related untrustworthy news retweeted by political communities in italy. considering the urgent need to address the societal problems caused by the spread of misinformation in italy, we created irma (the italian corpus of misinformation), a corpus containing over 600,000 italian news articles scraped from websites classified as untrustworthy sources by professional fact-checkers. ## method we decided to use source trustworthiness assessment as a proxy to identify the material of interest (cf. @xcite . therefore, we opted for two different misinformation databases, namely newsguard (ng, newsguard, 2020) and the misinformation domains (md) dataset foot_1 . ng is a professional factchecking database that provides indexes of trustworthiness for thousands of news domains. it rates domains in several categories related to news transparency and journalism ethics. the md dataset is an open-source collection of domains referenced by @xcite and extended with other lists curated by fact-checking collectives, individual scholars, and journalists. we decided to use two different databases in order not to be too dependent on one individual source. we chose these two datasets since, differently from other misinformation databases, they comprise a considerable number of italian sources. we also opted for domainbased rather than article-based fact checking to present a greater variety of data. we recognise that an article-based fact-checking service could have improved the "precision" of the material provided; however, we also think that domain-level fact-checkers represent an optimal balance between quantity and quality of (mis)information. we agree that not all sources deliver only unreliable news. however, having varying degrees of misinformation is an advantage. future studies could manually annotate documents in irma to offer a finegrained indicator of misinformation, helping the development of classifiers @xcite . finally, the growing number of scholars who have used these two fact-checking databases attests to their reliability (e.g., @xcite @xcite ). ## exploring irma's features in this section, we explore some of irma's features and provide examples replicating previous works. we checked whether the type of incoming traffic (i.e., direct and search) was related to websites' credibility, as previous works show @xcite . we found that credibility of websites was related positively with search traffic (@xmath0.44, @xmath1.0016) and negatively with direct traffic (@xmath2.30, @xmath3.0341), suggesting that con-firmation bias drives traffic towards towards misinformation websites also in this italian sample. we also tested the degree to which credibility was linked to interconnectedness, that is how multiple ideas form a dense and highly interconnected network, a property of conspiracy narratives @xcite . to this purpose, we created networks for each website from the cooccurrence of the top-fifty most frequent words extracted from the tf-idf (see variable tfidf10 in finally, we explored to what extent lexical features were linked to websites' credibility. to this goal, we fitted a series of linear models predicting credibility by the lexical features extracted with liwc. in figure 2 , we show the 20 highest and 20 lowest beta coefficients from regression (all @xmath7.001, bonferroni corrected). results parallel previous works @xcite @xcite showing that low quality sources tend to endorse a language characterized by anger (category rabbia), negative emotions (emo_neg), causality (causa), and negations (e.g., "do not", negazio) along with use of longer words (indexing sophisticated lexicon, bigwords), swear words (parolac), and longer texts overall (i.e., word count wc). ## conclusions we introduced irma, our publicly available corpus of 'untrustworthy' news in italian. this is, as far as we know, the first italian corpus of its kind. it consists of over 600,000 texts (335+ million words) and a number of variables to help scholars find the material that meets their needs. it can be used to develop deep learning classifiers as well as conduct different types of qualitative/quantitative research. irma allows for a vast range of textual analyses thanks to the variety and quantity of data and -0 . 1 0 . 0 0 . 1 0 . 2 r e li g io m e t a f is s e n _ p o s s o c ia l r if _ g e n f a m ig l e m o _ p o s n o i a lt r i o t t im is p r o n o m i s e i n c lu s i t u c o m m a a f f e t t i o t e m p o w p s c e r t e z a r t ic o l n u m e r o c a u s a e x c la m p a s s a t o p a r o la c i n ib iz e s s e r e n e g a z io l u i_ v e r b l a v o r o b ig w o r d s e m o _ n e g a ll p u n c w c p e r io d l o r o _ v e r r a b b ia o t h e r p s o ld i beta coefficient figure 2: coefficients (β, y axis) from regressions predicting liwc lexical features (on the x axis) by websites credibility scores. positive values indicate the feature being positively correlated with credibility. despite the fact that the two datasets label the websites in irma as "untrustworthy", this does not necessarily imply that they are all actively spreading fake news. this is due to the fact that domains could be rated not just on news quality and reliability, but also on other complimentary factors such as company policies (e.g., whether and how websites disclose information about ownership and financing). however, we are unable to provide the classification standards for the domains in our database, as well as the domains themselves due to restrictions of ng proprietary data policies. therefore, we suggest prospective users to judge the quality of news for themselves perhaps via data-driven approaches. finally, it is important to note that both the ng and the md datasets can vary over time, thus websites previously deemed untrustworthy may no longer be so in the future (and vice-versa).