“We Demand Justice!”: Towards Social Context Grounding of Political Texts (2024)

Abstract: Political discourse on social media often contains similar language with opposing intended meanings. For example, the phrase thoughts and prayers, is used to express sympathy for mass shooting victims, as well as satirically criticize the lack of legislative action on gun control. Understanding such discourse fully by reading only the text is difficult. However, knowledge of the social context information makes it easier. We characterize the social context required to fully understand such ambiguous discourse, by grounding the text in real-world entities, actions, and attitudes. We propose two datasets that require understanding social context and benchmark them using large pre-trained language models and several novel structured models. We show that structured models, explicitly modeling social context, outperform larger models on both tasks, but still lag significantly behind human performance. Finally, we perform an extensive analysis, to obtain further insights into the language understanding challenges posed by our social grounding tasks.

## introduction over the past decade, micro-blogging websites have become the primary medium for us politicians to interact with general citizens and influence their stances for gaining support. as a result, politicians from the same party often coordinate the phrasing of their social messaging, to amplify their impact @xcite . hence, repetitive, succinct phrases, such as "thoughts and prayers", are extensively used, although they signal more nuanced stances. moreover, the interaction among politicians from opposing parties often leads to messaging phrased similarly, but signaling opposing real-world actions. for example, 'thoughts and prayers', when used by republicans, expresses condolences in mass figure 1 : an example of varied intended meanings behind the same political message depending on the author and event in context shooting events, but when used by democrats conveys an angry or sarcastic tone as a call for action demanding "tighter gun control measures". similarly, fig. 1 shows contrasting interpretations of the phrase "we need to keep our teachers safe!" depending on different speakers and in the context of different events. humans familiar with the stances of a politician and, possessing knowledge about the event from the news, can easily understand the intended meaning of political phrases. however, automatically understanding such language is challenging. our main question in this paper is -can an nlp model find the right meaning? from a linguistic perspective, we follow the distinction @xcite between semantic interpretation (i.e., meaning encoded directly in the utterance and does not change based on its external context), and pragmatic interpretation (that depends on extra-linguistic information). the latter has gathered significant interest in the nlp community recently @xcite , focusing on language understanding, when grounded in an external context @xcite . to a large extent, disambiguation: the withdrawal from the paris climate agreement is the first step of many to come for the trump administration. it will not be the last, as more positive changes are sure to follow. ## social context grounding tasks we design and collect two datasets for social context grounding evaluation, and define three pragmatic interpretation tasks. in the tweet target entity and sentiment dataset, we collect annotations of opinionated tweets from known politicians for their intended targets and sentiments towards them. we focus on three political events for this task. the dataset and its collection are described below in section 3.1. in the vague text disambiguation task, we collect plausible explanations of vague texts, given the social context, consisting of author affiliation and specific event. we focus on eight political events. this dataset is detailed in section 3.2. examples and data statistics are shown in ## modeling social context the key technical question this paper puts forward is how to model the social context, such that the above tasks can be solved with high accuracy. we observe that humans can perform this task well (section 6.3), and evaluate different context modeling approaches in terms of their ability to replicate human judgments. these correspond to no context, text-based context representation (e.g., twitter bio, relevant wikipedia articles), and graph-based context representation, simulating the social media information that human users are exposed to when reading the vague texts. we report the results of all our baseline experiments in we evaluate gpt-3 foot_2 in zero-shot and four-shot in-context learning paradigm on both tasks. we provide contextual information in the prompt as short event descriptions and authors' affiliation descriptions. note that gpt-3 is trained on news data until sep. 2021 which includes the events in our data (models denoted llm baseline). we evaluate the performance of politician embeddings from political actor representation (par) @xcite and discourse contextualization framework (dcf) @xcite ## results the results of our baseline experiments are described in tab. 2 and 3. we evaluate our models using macro-averaged precision, recall, f1, and accuracy metrics (due to class imbalance, we focus on macro-f1). several patterns, consistent across all tasks, emerge. first, modeling social context is still an open problem. none of our models were able to perform close to human level. second, adding context can help performance, compared to the no-context baselines, models incorporating context performed better, with very few exceptions. third, llms are not the panacea for social-context pragmatic tasks. despite having access to a textual context representation as part of the prompt, and having access to relevant event-related documents during their training phase, these models underperform compared to much simpler models that were fine-tuned for this task. finally, explicit context modeling using the dcf model consistently leads to the best performance. the dcf model mainly represents the social context in the form of text documents for all nodes. further symbolic addition of other types of context such as social relationships among politicians and relationships between various nodes could further help in achieving better performance on these tasks. in the target entity task, roberta-base + dcf embeddings obtain 73.56 f1 vs. 68.83 for the best no-context baseline. twitter bio and wiki-context hardly improve, demonstrating the effectiveness of modeling contextual information explicitly vs. concatenating context as text documents. no context performance well above the random performance of 50 f1 indicates the bias in the target entity distribution among classes. we discuss this in section 6.4. in sentiment identification task, we see that bert-large + dcf back-propagation outperforms all other models. vague text disambiguation task results in ## conclusion and future work in this paper, we motivate, define, and operationalize 'social context grounding' for political text. we build two novel datasets to evaluate social context grounding in nlp models that 'are easy for humans' when the relevant social context is provided. we experiment with many types of contextual models. we show that explicit modeling of social context outperforms other models while lacking behind humans.