An Interactive Framework for Profiling News Media Sources (2024)

Abstract: The recent rise of social media has led to the spread of large amounts of fake and biased news, content published with the intent to sway beliefs. While detecting and profiling the sources that spread this news is important to maintain a healthy society, it is challenging for automated systems.In this paper, we propose an interactive framework for news media profiling. It combines the strengths of graph based news media profiling models, Pre-trained Large Language Models, and human insight to characterize the social context on social media. Experimental results show that with as little as 5 human interactions, our framework can rapidly detect fake and biased news media, even in the most challenging settings of emerging news events, where test data is unseen.

## introduction the recent rise of social media has enabled information to spread at a rapid pace, having the potential to very quickly impact a large number of people, especially during key events, such as political elections @xcite . while this rise of social media has many benefits, one downside is that harmful information, i.e. fake or politically biased news, can also spread rapidly, affecting people's perspectives. thus, detecting it is important. while one approach is to fact-check or detect the bias of all content on social media (i.e. twitter), another is to focus on the source, and ask: is this source factual or politically biased? this task, profiling news media sources, which we focus on, can scale better, as often times a large amount of the content sources' publish have the same factuality/political bias as the source itself. we model this on a 3-point scale: high, low, and mixed factuality, and left, center, and right bias. details: app. a.1. even at the source level, it is difficult for humans to profile all news content, due to the large number of sources online. further, this task is still challenging for ai systems @xcite @xcite , especially in the emerging events settings, when the system is tested on its ability to adapt to new events, consisting of new sources, content they generate, and social media users engaging with them that were not seen at training time @xcite . for example, in a graph framework, test set nodes are not connected to training set nodes. in these settings, large language models (llms) also struggle, even when enhanced with extra knowledge @xcite . due to the struggles of ai systems to automatically profile news media, in this paper we propose a different, interactive approach, for this task. we are inspired by recent results @xcite showing that misinformation and highly biased content tends to spread in closely knit communities on social media. this leads us to ask, whether better modeling of the social media relationships that underlie content spread would improve our ability to profile the content itself. specifically, we hypothesize that users on social media form information communities, or groups of users, where certain themes circulate more in some communities vs. others. if we can identify these themes and use them to identify information communities, then we can better profile the content discussed by the communities. for example, a user joining a left-leaning community, is more likely to be left-leaning, and so is the content they share. then, if that user shares content from a source, that source is also more likely to be left-leaning. in the settings of emerging events, we must be able to characterize and form the information communities quickly, without using labeled training data, so we can rapidly detect fake/biased news sources on unseen data. unfortunately, using only information on social media to form information communities involves complex reasoning on unseen data, and is thus challenging for automated from the learned graph model (b), we find candidate information communities through k-means clustering. using a llm, gpt-3, we form a textual representation of the information community by summarizing its users, and then ask humans to narrow down the community based on the user summaries (d), forming smaller, validated communities, whose users are then connected to each other. we then expand the validated communities, by again model clustering users, forming user summaries, but this time asking gpt-3 to place or not palce the users into validated communities, which can be done repeatedly (e). this entire process (c-e) can repeat, starting with clustering of unassigned users (c), to form more validated communities, which can be expanded further. specifically, to form the communities, we first form initial candidate communities by clustering social media users based on their graph model embeddings, capturing model beliefs. as news media profiling is difficult, these candidate information communities are likely to be imperfect, i.e., some users are likely to be inconsistent with their community assignment. thus, these assignments need to be validated, by examining their actual information preferences and the content they generate, rather than using only relational information. this would help ensure that only users that are actually similar to each other (i.e. have similar perspectives on similar content) make up each community. we use llms and human interaction to do this validation. first, we summarize each user in the community using llm's, mapping their profile to a short statment capturing their views. this representation facilitates the human validation step, in which we ask humans to determine which users are similar, and by that form validated communities (experiments show this is hard for llms, but simple for humans). we then expand the validated communities, adding new users to them, based on model clustering. these assignments are again validated based on users' summaries, but this time using llms to compare new users to users in human validated communities (we use human interactions as training examples for few-shot user similarity detection, a simpler llm task). for users not assigned to validated communities, the above process is repeated, expanding the # of communities. fig. 1 shows an overview. in short, humans interact to help form initial communities, which are then automatically expanded using graph and llm similarity. in summary, we make the following contributions: (1): we formulate the task of interactive news media profiling, by presenting a framework to build information communities. (2) we take advantage of graph model, llm, and human knowledge to perform this interactive task rapidly. (3) we evaluate on one of the most challenging news media profiling settings, emerging news events, showing how our interactive framework leads to performance improvements with less than 5 human interactions. more generally, our framework can be used to rapidly profile news media, without any additional labeling and minimal human effort. sec. 3 describes our graph, sec. 4 our interactive framework, sec. 5 results, and sec. 6 analysis. ## related work over the last few years, there has been a large interest in profiling news media. @xcite proposed datasets for political bias detection, while @xcite @xcite study methods for it. further, fake news detection has also been a hot research topic, studied in graphs @xcite @xcite , cross-domain @xcite @xcite , and low-resource @xcite settings, amongst others. we focus specifically on news media profiling in the emerging events setting, which is extremely challenging, as test data is unseen and does not interact with any train data. thus, this setting is ideal for us to measure the benefits of human interactions. it also has received a lot of recent attention due to its' challenge @xcite , and we hypothesize that some of these methods can be combined with our work. large language models (llms) have been applied to many tasks, as they can capture lots of knowledge @xcite . however, llms still cannot reason well, and thus struggle on harder tasks like fake news detection @xcite . we instead use llm's successful properties to amplify the impact of human interactions. interactive ml has also been studied and applied to many tasks @xcite @xcite @xcite @xcite . building information communities is also a popular research area, whether it be through embeddings or dnn modeling approaches @xcite . prior work also shows misinformation spreads in communities @xcite . we propose to interactively build information communities, by humans interacting with llms and graph models. we discuss more related works in app. e. ## graph overview in this paper, we focus on detecting the political bias and factuality of news media sources, which we call news media profiling. we model sources for factuality and political bias on a 3-point scale: low, mixed, and high factuality, or left, center, and right political bias. more details about this task setup and its importance are in @xcite and appendix a.1. we use the public graph-based social media analysis model from mehta et al. foot_0 , which they trained for fake news source detection. as we also evaluate news source bias detection, we train the graph for both objectives, on twitter data. we refer to @xcite for the details of this graph model, but briefly explain it here. sec. 4 explains our interactive protocol for identifying information communities. the model uses a heterogeneous graph, encoded using relational graph conv. networks (r-gcn), to capture the relationships between sources, articles, and users. based on the r-gcn representations, factuality and bias of news sources can be predicted. r-gcns allow us to better capture relationships in the graph, such as a source represented in part by the users that follow it (which are also represented by their relationships to other nodes). graph creation using twitter social context: our graph (fig. 1a , mehta et al.) consists of 3 node types: news sources (s), articles (a) they publish, and twitter users (u ) that interact with sources and articles. we connect sources to articles they publish, via edges. further, users are connected to sources and other users they follow, and articles they propagate (retweet or tweet the link of). the twitter users provide the social information in the graph, which we later aim to better learn. graph training using social context: similar to mehta et al., we train a r-gcn @xcite to learn the graph. we train the classification objective of both fake news source detection and news source bias detection, using a separate fully connected (fc) layer for each, optimizing them jointly by summing the losses. once the model is trained, we can use it to obtain meaningful node embeddings for every node in the graph, and profile news sources. more details: app. a.3. ## interactive approach while the above graph-based model proposed by @xcite achieves strong performance on fake news source detection when evaluated in transductive settings (test data seen at training time), it struggles in the fully inductive settings (test data unseen), and in general performs well below human baselines. we thus propose an interactive approach, combining the strengths of graph models, large language models (llms i.e. gpt-3), and humans, to profile content on social media better. our approach hinges on the fact that if we can better model user preferences and thus user similarity, we can better model the content they fakenews_inference_operators propagate. this is because, similar users are likely to have similar interests, and thus share similar content, which in turn is likely to have similar levels of factuality/bias. for example, a group of users sharing content in support of lowering taxes and decreasing regulations are more likely to be rightbiased (i.e. republican) vs. left-biased (i.e. democrat), and thus any source content they share is also more likely to be right-biased. thus, we hypothesize that the larger the groups of users with similar content preferences we can form, the higher our performance is likely to be. further, if we explicitly create new graph edges between these similar groups of users, this information will flow to other users not part of these groups, and eventually news sources, increasing classification performance. fortunately, while modeling user content preferences solely through ai models like llms is difficult @xcite , humans can quickly determine if two users are similar, forming an initial group. then, we hypothesize, that llms can be prompted using the human insight to extend the group, by asking them if other users are similar in the same ways. thus, in this paper, we propose an interactive framework, taking advantage of human and llm strengths to better model user content preferences, and improve media profiling. specifically, in this sec., we discuss the interactive approach we propose to form these groups of similar users, or information communities. we take advantage of the strengths of trained graph model knowledge, llm knowledge (gpt-3), and human insight to design an iterative, interactive approach: we first use the graph models' learned user similarities to form initial candidate information communities (sec. 4.1), which are summarized by llm's (sec. 4.2), and then validated by a human interactor (sec. 4.3). these validated communities are then expanded upon to include more users in sec. 4.4, by again using graph model knowledge and llm's. however, this time llm's are prompted based on the human interaction, to validate user assignments to communities. this expansion step can be done iteratively, i.e. assigning additional users to validated communities. once enough users are not assigned to existing communities, we form an additional set of human validated communities, repeating the the above process from sec. 4.1-sec. 4.4, and the number of total validated communities is increased (sec. 4.5). finally, once enough information communities are formed, we can then learn them (sec 4.6), updating graph model parameters. we later show that through this iterative process, minimal human interactions can lead to significant performance improvements for news media profiling. fig. 1 and alg. 1 shows an overview. algorithm 1 our interactive framework to find validated information communities 1: input: u (users), ue (graph user embeddings), v (empty list to store validated information communities) 2: output: v (validated communities) 3: iteratively find information communities 4: while not converged do 5: c 1...@xmath0: c1, @xmath1...k )) choose the highest purity clusters, that discuss the same entity 7: s1, @xmath2, c2) use gpt-3 to summarize the users in each cluster 8: v.append([human(s1), human(s2)]) validate clusters using humans to form validated communities 9: now, iteratively expand each validated community 10: while not converged do 11: c 1...@xmath3: c 1...@xmath4...k , v ) for each cluster, find the m nearest neighbors to each validated community, that's our new cluster 13: @xmath5...k , v )) for each cluster, ask gpt-3 to assign or not assign users to validated clusters, expanding them 14: end while 15: end while 16: return v (validated information communities) ## discussion in this section, we evaluate our black lives matter interactive framework (sec. 5.5) learned information communities. we begin by analyzing the cohesiveness of the communities, first human (sec.. 6.1) and second automatically (app. f.1). we then show why human interactions are critical (sec. 6.2). finally, we analyze the communities themselves, analyzing the topics discussed (app. f.2). ## conclusion in this paper, we proposed a framework for interactive news media source profiling. our framework combines the strengths of graph based news media profiling models, llms, and humans, to build stronger information communities. we show how without any additional labeled data, and less than 5 human interactions which can be done in under 10 minutes, we can better detect fake news and bias sources, on two separate news events, even in the most challenging setting of emerging news. our future work is building larger and better communities, and having more human interaction rounds. we hypothesize that more data (users, sources, articles, and human interactions) could lead to lead to better communities, as our approach can capture more perspectives on social media. this would also likely lead to a better trend in the results, leading to more consistent performance improvements as the number of interaction rounds are increased.