Variation between credible and non-credible news across topics (2024)

Abstract: ‘Fake News’ continues to undermine trust in modern journalism and politics. Despite continued efforts to study fake news, results have been conflicting. Previous attempts to analyse and combat fake news have largely focused on distinguishing fake news from truth, or differentiating between its various subtypes (such as propaganda, satire, misinformation, etc.) This paper conducts a linguistic and stylistic analysis of fake news, focusing on variation between various news topics. It builds on related work identifying features from discourse and linguistics in deception detection by analysing five distinct news topics: Economy, Entertainment, Health, Science, and Sports. The results emphasize that linguistic features vary between credible and deceptive news in each domain and highlight the importance of adapting classification tasks to accommodate variety-based stylistic and linguistic differences in order to achieve better real-world performance.

## introduction the term 'fake news' catapulted to popularity around the 2016 u.s. presidential election and has continued to cast a shadow of mistrust over journalism and politics @xcite . global trust in social media as a news source remains low and has been on a decline for the past decade @xcite . despite this, many still turn to social media as a means to stay informed. half of the u.s. adult population report getting their news from social media at least some of the time @xcite . however, most users express concerns about quality, accuracy, and bias @xcite . the effort to combat the spread and influence of 'fake' or 'non-credible' news has been reflected in the large body of academic research on fake news detection and analysis. however, there has been little large scale practical implementation of this research. in part, this can be attributed to conflicting observations in the literature. this paper takes a variety specific approach to non-credible news analysis by investigating linguistic and stylistic differences for five common news topics: economy, entertainment, health, science/technology, and sports. ## related work deceptive and persuasive language: in political communication, advocates attempt to manipulate the public in many ways. arguments can be classified into four types depending on whether it is pro, con, easy, or hard to comprehend @xcite . to assess the persuasive power of each argument type, @xcite studied opinions on the north american free trade agreement (nafta) and healthcare at three points in time. oppositional arguments held more weight, and for nafta the effect was stronger for hard arguments. however, easy arguments were more persuasive for healthcare. in policy proposal, @xcite observed that persuasion can be influenced by the formulation and presentation of interpretations. an argument is more persuasive, regardless of a voter's political beliefs, if one can control the environment to allow for only one interpretation. studies on lying have revealed that certain cues can be used to indicate deception. it has been shown that the psychological burden of lying, whether due to guilt or the challenge of remembering the lie, may cause liars to avoid language that takes ownership of the statement or portray certainty @xcite @xcite . newman et al. @xcite found that, in addition to using more words that elicit negative emotion, liars distance themselves from claims by using fewer first and third person pronouns. in a study of true and false statements in polish and english, @xcite also observed that lying in en-glish triggered an increase of words with negative tone, as well as a general increase in negation. deception in news text: while there have been several studies investigating linguistic and stylistic features in news text, results are often contradictory. @xcite used style analysis, including readability scores and dictionary features, to distinguish hyper-partisan from mainstream news. it was found that left and right-wing news share more stylistic similarities with each other than with mainstream. writing style on its own was discovered to be sufficient for distinguishing hyper-partisan news articles from more balanced news. @xcite found that proper nouns and passive voice is more frequent in credible news, while non-credible news uses more superlatives. while identifying linguistic features to use in automatic classification, kasseropoulos and tjortjis (2021) noted that fake articles are shorter in length and use fewer technical words, quotes, punctuation, and have more lexical redundancy. they also use simpler language with shorter words, as well as more personal pronouns and adverbs. in satire, surface level features such as sentence length and average word frequency, in addition to semantic features and causal connectives vital to text comprehension are considered predictors @xcite . disinformation is also prone to grammatical and orthographical mistakes, erratic punctuation, and idiosyncratic typography @xcite . another study investigated variation between credible and deceptive hard news and found that credible news was more informationally dense, while deceptive news was more narrative @xcite . it was also observed that credible news used more adjectives, intensifiers, and clausal coordination. @xcite used interpersonal deception theory (idt) and reality monitoring (rm) to analyse the language used by russian trolls during 2016 u.s. presidential election. they identified 49 linguistic cues used by deceivers which indicated uncertainty, including hedges, modal verbs, auxiliary verbs, and expressions of possibility. it was also observed that deceivers attempted to distance themselves from the lies by using less selfreference in the form of pronouns. in research on climate change news and editing of international news, a trend of misreporting illustrated by errors such as overstatement, misquotation, misattribution, and over-assertion, was revealed @xcite . it also appears that differences between fake and credible news vary across languages and dialects. in a study of english and portuguese fake news texts, sousa-silva (2022) noticed that variation between fake news and mainstream media differ depending on the corpus. the longest words are used by fake news in the english corpus, whereas the longest words are used by mainstream media in portuguese. in a study of false statements in brazilian portuguese news, @xcite used word and sentence level analysis to discover that true statements used more nouns and verbs, while false statements were found to contradict previous literature in their pronoun usage. in depth discourse and linguistic analysis of deceptive text has also revealed some interesting trends throughout the various forms of fake news. using van leeuwen's discourse model of legitimation and de-legitimation, igwebuike and chimuanya (2021) analysed the legitimation strategies used for justification of fake news posts on nigerian whatsapp, facebook, and twitter. findings revealed that creators convey messages to readers and validate disinformation through appeals to authority, emotion, moralisation, and rationalisation. deception detection: @xcite used linguistic features to classify the veracity of news content by organizing the features into sets and merging them with word embeddings. the 20 most salient features were selected and applied to a voting classifier. @xcite employed 16 linguistic features categorized into four classes in a decision tree algorithm, achieving an accuracy of 60.72%. @xcite used a variety of features from text (e.g. number words, sentences, and characters), along with user and message specific features to identify hoaxes and fake news on social media with various machine learning models. introducing the small novel unbiased dataset, @xcite tested 57 linguistic features embedded with word-to-vector embedding in several popular ml classifiers for deception. kasseropoulos and tjortjis (2021) identified an optimal set of 23 features out of 87 which performed well with cnn and lstm classifiers. lux (language under examination), is a text classifier that makes use of linguistic analysis to infer the likelihood of an input being fake-news @xcite . linguistic metrics were included as model features to improve classification performance in identifying fake news. many other approaches to automatic deception detection have made use of shallow text features and semantics in their models with reasonable success in the task at hand @xcite . @xcite used models trained on bag-of-n-grams and bag-of-rst (rhetorical structure theory) features to detect satire, real, and fake news in russian and discovered that unigrams were the most important feature for detection. in a survey of supervised learning approaches to deception detection with discourse and structural features, the results of such approaches were mixed, but showed promise @xcite . the results from previous literature show a conflicting landscape of deceptive language in writing and news text. likely due to this inconsistency, automatic deception detection methods which have utilised linguistic features exclusively typically achieve lukewarm performance. one of the limiting factors of the previous approaches is that analysis has been broadly based on hard news and politics with little investigation into other news topics. the research presented in this paper adds to the existing body of literature by including underrepresented news topics, such as entertainment and sports. the results of this paper will reference the previous findings detailed above, identifying similarities and differences. the following sections will introduce the data utilised in this study and the methodology through which they have been analysed. ## data a novel dataset was created in order to ensure a balanced sample size with consistent topic labelling. texts are limited to english news from the united states and canada during the period of 2011 to 2018. article annotation was carried out automatically as data was collected. articles were labelled 'non-credible' or 'credible' based on publisher intent, similar to the approach taken by @xcite . articles from publishers whose mission is perceived as providing accurate information with high reliability are categorized as 'credible', whereas articles from publishers who intentionally produce fabricated stories or have mixed/lower factuality ratings are categorized as 'non-credible'. labels are determined based on bias and reliability scores provided by media bias-fact check, foot_0 allsides, foot_1 and ad fontes media. 3 bias ratings are determined using a numerical scale, based on various factors (including political leaning, factuality, spin/framing, and several types of bias), averaged from a survey of articles from the outlet. while these companies have slightly different approaches to rating, all employ a panel based system where a selection of articles and headlines from an outlet is reviewed regularly by a balanced panel of raters who have self reported their political biases. as these organisations are private companies, specific guidelines are not publicly available. however, assessment criteria are described in detail on the respective websites. the labels 'credible' and 'non-credible' were chosen based on the definition of 'credibility' as something trustworthy or worthy of belief. this label covers the range of deceptive topics in the analysis, including satire. despite the primary intent of satire being entertainment, it is still considered noncredible due its potential to mislead readers and its lack of trustworthiness as a source of information. furthermore, while automatic classification between hard news and satire has been somewhat successful @xcite , it is often challenging to distinguish satire from other forms of deceptive news @xcite . sources for the credible corpus are reuters, the new york times, global news, business insider, cbc, and the new yorker. non-credible news sources include the beaverton, breitbart, global research, if you only news, your newswire, mad-world news, and liberty writers. there is a total of 41,172 articles in the combined non-credible and credible news corpus, with a 50:50 split of deceptive and credible news for each topic. ## conclusions and future work although differences between credible and noncredible news were observed in all topics, details varied considerably. importantly, while not all cues of deception identified in previous literature were present in every domain, most topics showed at least one characteristic of deceptive language. plank (2016) argues that many nlp models suffer when applied to the real world because they are trained on canonical data. while there appears to be some characteristics of deceptive news text that are shared, primarily technical language, topic differences between credible and non-credible news are too varied for tasks involving canonical 'fake news'. for non-credible news classification tasks, it is beneficial to focus on adapting approaches to specific topics. appeals to emotion and language that elicits negative emotion have been identified as features of deceptive language and text @xcite @xcite . in the future, it will be useful to investigate negativity in non-credible news topics by using psycholinguistic features with linguistic inquiry and word count (liwc). in light of recent technological advancements, it would also be interesting to compare llm generated noncredible news to see if features of deception are also present in generated news. further research may also look into stylistic differences in news from other regions as deception cues are likely to vary based on culture and language.