Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting (2023)

Abstract: Automatic response forecasting for news media plays a crucial role in enabling content producers to efficiently predict the impact of news releases and prevent unexpected negative outcomes such as social conflict and moral injury. To effectively forecast responses, it is essential to develop measures that leverage the social dynamics and contextual information surrounding individuals, especially in cases where explicit profiles or historical actions of the users are limited (referred to as lurkers). As shown in a previous study, 97% of all tweets are produced by only the most active 25% of users. However, existing approaches have limited exploration of how to best process and utilize these important features. To address this gap, we propose a novel framework, named SocialSense, that leverages a large language model to induce a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. We hypothesize that the induced graph that bridges the gap between distant users who share similar beliefs allows the model to effectively capture the response patterns. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings, demonstrating its effectiveness in response forecasting. Moreover, the analysis reveals the framework’s capability to effectively handle unseen user and lurker scenarios, further highlighting its robustness and practical applicability.

## introduction the code is available at https://github.com/ chenkaisun/socialsense prediction model sentiment polarity sentiment intensity user pro le social context historical posts -[reply] how funny! democrats screwing democrats over. -[reply] you mean like your administration did along with the illegal fisa warrants -[post] just do it...... proud american willing to consider all political & religious views. defender of democracy-not political correctness. no dms w/solicitation. research that enables numerous influential applications, such as offering content producers a way to efficiently estimate the potential impact of their messages (aiding the prevention of unexpected negative outcomes) and supporting human writers in attaining their communication goals @xcite for risk management. this direction is especially important nowadays as the proliferation of ai-generated misinformation, propaganda, and hate speech are becoming increasingly elusive to detection @xcite . in this context, accurately forecasting the responses from different audiences or communities to news media messages becomes critical. one of the primary challenges in personalized response forecasting lies in developing effective user representations. a crucial aspect to consider when representing a user is the integration of social dynamics (e.g., social interactions around a user) as well as their individual beliefs and interests. this becomes particularly relevant for users who lack explicit profiles or historical activities (commonly referred to as lurkers). previous efforts, however, have yet to explore the types of structural information that are helpful and how to best utilize such information @xcite @xcite . during our preliminary analysis, we observed that users who share similar beliefs, specifically social values, are often situated in distant communities within the explicit social network. to provide further context, our findings reveal that a significant portion (over 44.6%) of users in the network data we collected for our experiment share beliefs with other users who are at least two hops away in the network. this emphasizes the importance of considering the connections between users with similar beliefs, even if they are not directly linked in the social network. furthermore, previous research has indicated that user history plays a significant role in the model's performance. however, it is often directly utilized without processing in existing approaches, leading to the introduction of noise in the modeling process. motivated by these findings, we introduce so-cialsense (where sense refers to the understanding and perception of social dynamics and behaviors within the online realm), a novel framework for modeling user beliefs and the social dynamics surrounding users in a social network. in this work, we conduct experiments using the socialsense framework in the context of response forecasting. our approach aims to capture the pattern of how "similar neighbors respond to similar news similarly". to harness the potential of network features, we curated a new user-user graph comprising 18k users from twitter (the data will be anonymized when released), augmenting the original dataset @xcite . the socialsense framework consists of three key stages: (1) inducing latent user personas using the large language model (e.g., chatgpt @xcite ), (2) building a belief-centered network on top of the existing social network, and (3) propagating information across multiple levels. we demonstrate the effectiveness of our method through experiments on the dataset from @xcite . our results show that our framework outperforms existing baselines consistently across metrics in both zero-shot and fully-supervised settings. we further conduct a detailed analysis to address research questions concerning the model's general-izability to unseen users and its predictive capabilities for lurkers. our findings reveal two additional key insights: (1) the model performs exceptionally well in scenarios involving lurkers, outperforming the baseline by over 10% accuracy score in sentiment polarity forecasting, and, (2) compared to baseline approaches, the model exhibits consistently better generalization capabilities when applied to unseen users. additionally, our analysis underscores the significance of various components within the belief-augmented social network, revealing that both the belief-centered graph and the user-news interaction network play vital roles in determining the network's overall performance. ## task formulation in the task of response forecasting on personas for news media, our objective is to predict how users will respond to news media messages. specifically, we focus on analyzing the sentiment intensity and polarity of these responses. formally, given a persona p (representing the user) and a news media message m, our goal is to predict the persona's sentiment polarity ϕ p (categorized as either positive, negative, or neutral) and intensity ϕ int (measured on a scale of 0 to 3) of the persona's response. we frame this task as a multi-class prediction problem. ## socialsense to accurately predict individuals' responses, it is crucial to develop an effective user representation that captures their personas. while previous studies have utilized user profiles and historical data to model individuals' interests with reasonable accuracy, there is a significant oversight regarding the behavior of a large number of internet users who are passive participants, commonly referred to as lurkers. this phenomenon is exemplified by statistics showing that only 25% of highly active users generate 97% of the content on twitter @xcite . consequently, the sparse historical data available for lurkers makes it challenging to infer their responses reliably. to address this issue, a social network-based approach can be employed to leverage users' social connections, gathering information from their neighbors. however, it is important to question whether relying solely on social networks is sufficient. in this work, we introduce a novel perspective by borrowing the concept of belief and defining it in terms of social values. by considering so- identity user information extraction human value moral value issues profession others latent persona neighbor 1 propagation neighbor n ... attributes operator llm-based propagation for zero-shot only ## related work existing research has focused on predicting the individual-level response using additional textual features as well as deep neural networks (dnn) @xcite @xcite . however, these existing methods neglected the important information about users' personas as well as the modeling of graph-structured interactions among users with the social items. another line of related works formulates the response forecasting as text-level generation task @xcite @xcite . however, these lack a quantitative measure for analyzing the response (such as in the sentiment dimensions), limiting their applicability in downstream tasks like sentiment prediction on impact evaluation of news @xcite . in contrast, we propose a novel framework that leverages large language models to induce the graph structure and integrates disentangled social values to forecast responses, whether in a supervised or zero-shot manner. our work demonstrates that effectively modeling the social context and beliefs of users provides a clear advantage in the social media response forecast task. this can ultimately benefit various downstream applications such as assisting fine-grained claim frame extraction @xcite and situation understanding @xcite . in the field of social-nlp, related research has focused on applying nlp techniques, large language models (llm), and prompting strategies to model, analyze, and understand text data generated in social contexts. for instance, progress has been made in misinformation detection @xcite @xcite and correction @xcite , propaganda identification @xcite @xcite , stance detection @xcite , ideology classification @xcite , lm detoxification @xcite , norms grounding @xcite , popularity tracking @xcite , and sentiment analysis @xcite @xcite . the emergence of advanced decoder language models like chatgpt has led to extensive research on prompting techniques and their application across various nlp tasks @xcite @xcite @xcite . indeed, experiments have shown that chat-gpt even outperforms crowd workers in certain annotation tasks @xcite . however, when it comes to social tasks like response forecasting, relying solely on large-scale models without taking into account the social context and users' personas may not yield optimal performance @xcite . our experiments demonstrate that incorporating social context in the prompt consistently enhances the llm's performance, as showcased in our simulation of information propagation using large language models. operates on the premise that connecting distant users in social networks facilitates the modeling of implicit communities based on shared beliefs. through comprehensive evaluations, we demonstrate the superior performance of our framework compared to existing methods, particularly in handling lurker and unseen user scenarios. we also highlight the importance of the different components within the framework. in future research, it would be valuable to explore the application of belief-augmented social networks in other domains and to develop an effective social prompting strategy for general-purpose applications. furthermore, it is worth investigating how response forecasting models can adapt efficiently to dynamically evolving data, especially given the swift changes observed in real-world social media platforms (de @xcite .