Towards Detecting Harmful Agendas in News Articles (2023)

Abstract: Manipulated news online is a growing problem which necessitates the use of automated systems to curtail its spread. We argue that while misinformation and disinformation detection have been studied, there has been a lack of investment in the important open challenge of detecting harmful agendas in news articles; identifying harmful agendas is critical to flag news campaigns with the greatest potential for real world harm. Moreover, due to real concerns around censorship, harmful agenda detectors must be interpretable to be effective. In this work, we propose this new task and release a dataset, NewsAgendas, of annotated news articles for agenda identification. We show how interpretable systems can be effective on this task and demonstrate that they can perform comparably to black-box models.

## introduction in recent years, the spread of misinformation and disinformation has become a particularly persistent and harmful issue online @xcite @xcite . for example, during the covid-19 pandemic in the united states, we saw several instances of malicious actors propagating disinformation regarding mask mandates, vaccines, and fake remedies and cures to discredit the government and public health officials. the people initiating these disinformation campaigns typically have some harmful agenda, such as discrediting an individual/group or encouraging disruptive realworld action. furthermore, with new conversational language models such as chatgpt and @xcite , a malicious actor can generate human-like harmful text content at scale. identifying these types of harmful news campaigns typically requires consideration of three important attributes: misinformation in news is any article which relies on false information and can therefore be identified by focusing on factuality. disinformation is deliberately misleading information created/disseminated with an intent to deceive @xcite , so can be identified by factuality and authorial deception. however, the degree of harm caused by disinformation and misinformation depends on the agenda (or goal) of the article. @xcite advocates for this kind of focus on agenda as a useful marker of intentionality in disinformation detection. defining what constitutes a harmful agenda is an inherently subjective task and requires a notion of good and bad. researchers in different domains have tried defining and formalizing the concept of harm, such as harmful online content @xcite , covid-19 related tweets @xcite , etc. however, to the best of our knowledge, the notion of harmful agendas in journalistic news articles has not been explored yet. in this paper, we therefore propose a new task of detecting harmful agendas in news articles. inspired by definitions of harm in other works, we specifically focus on real-world harm, meaning articles that spur core belief change or actions that significantly harm someone. to develop an initial detector, we formulate this task as classifying an article's agenda as harmful or benign, based solely on the article text, and we annotate a dataset, newsagendas, to evaluate performance. we note that future work could also formulate this problem in several other ways, such as also identifying the target audience, or additionally using metadata or contextual cues such as author information, publication platform, etc. we imagine this type of agenda detector could be used to flag potentially harmful articles for further inspection. it is therefore critical that any such detector be interpretable so that further examination could quickly reveal why an article was flagged and screen out any falsely identified articles. for sensitive application areas, there is a need to build models that are interpretable by design, rather than trying to interpret their results after the fact @xcite . given the subjectivity and the sensitivity of this task, we build an interpretable model that uses extractive rationale-based feature detectors to ensure faithfulness and interpretability, not only at the feature level but also at the text level. our primary contributions are: ## newsagendas dataset in order to evaluate our model's performance and contribute an initial benchmark for this task, we annotated news articles which we are releasing as a novel dataset, newsagendas. ## methods we leverage large weakly labeled datasets to train feature classifiers for our features of interest. we prioritize exploring different levels of interpretability in the models we compare and what performance tradeoffs come at each level. to focus our analysis, we select 7 features to study in-depth: clickbait, junk science, hate speech, conspiracy theories, propaganda, satire, and negative sentiment. out of the 4 features we excluded, 3 did not have enough labelled data. for the 4th, political bias, after consulting our journalism experts, we determined the relationship between harmful agendas in news articles and political bias is nuanced and needs further study. we therefore leave political bias to future work to promote simplicity and interpretability in our approach. ## results we investigate a series of research questions that analyze the efficacy of our overall approach, as well as individual components in our dataset and models. ## related work disinformation and misinformation. there are many previous approaches which have studied detection of misinformation and disinformation and which would be useful in combination with the detectors developed in this work (e.g., an agenda detection system flags an article to then go through a fact-checking pipeline). research on detecting fake news includes detectors based on linguistic features @xcite , fact-checking based systems @xcite , social context or propagation network based approaches @xcite @xcite , multimodal approaches @xcite , etc. other work has focused on characterizing/defining disinformation as a whole and developing classification schemas for campaigns @xcite . however, neither disinformation detection nor characterization has explicitly looked at the more specific identification of a harmful agenda in an article. intent detection. an agenda requires intention so detecting a harmful agenda is a type of intent detection. intent detection is used in many settings with systems using slot-filling @xcite , conversational techniques @xcite , and language understanding @xcite . there has also been research into what intentions are involved with news articles specifically -on the intention of writing vs. sharing articles @xcite , the journalis we also highlight the fresh-rationale words in the article opening (the title and first couple sentences) for clarity. ## conclusion in this work, we formalize the open challenge of detecting harmful agendas in news articles, release an initial evaluation dataset, and develop an inter-pretable system for this task. we hope our work can encourage future investment in this area -such as exploring state-of-the-art intepretable models for detecting the features we discussed, further characterizing article agenda beyond a binary classification, or investigating the interplay between text features and metadata like article source. to do harm is seldom an objective factor and is also difficult to measure or quantify. our experiments on inter-annotator agreement use a small dataset, so this study could be expanded with collaboration with social science researchers to better qualify how people perceive the agenda in different articles. our work is also grounded in the united states, so it may have limited applications to the news in other countries (discussed more in section 8). secondly, our data and framework can be used to build and train a system to perform posthoc detection of harmful agendas in news articles. however, in a real-world system, this identification would likely need to happen on the fly, so as to make readers aware of these agendas as they are exposed to the articles. finally, another aspect that we have not addressed in this study is the effect that a platform or community may have on the perceived harm in an article. for example, on dedicated social media channels hosting discussions on alternate theories and contentious topics (such as the efficacy of covid-19 vaccines), a junk science article with dubious claims may not be as "harmful" as opposed to the same article being posted on an open forum where readers may perceive it as scientific fact, thereby making the article more "harmful". the context in which news articles are disseminated may have a profound impact on this perceived harm and this may be an interesting direction for future exploration.