Probing Power by Prompting: Harnessing Pre-trained Language Models for Power Connotation Framing (2023)

Abstract: When describing actions, subtle changes in word choice can evoke very different associations with the involved entities. For instance, a company ‘ employing workers’ evokes a more positive connotation than the one ‘ exploiting ’ them. This concept is called connotation . This paper investigates whether pre-trained language models (PLMs) encode such subtle connotative information about power differentials between involved entities. We design a probing framework for power connotation, building on (CITATION)’s operationalization of connotation frames . We show that zero-shot prompting of PLMs leads to above chance prediction of power connotation, however fine-tuning PLMs using our framework drastically improves their accuracy. Using our fine-tuned models, we present a case study of power dynamics in US news reporting on immigration, showing the potential of our framework as a tool for understanding subtle bias in the media.

## introduction an author's choice of words evokes associations and reactions from a reader that go beyond the literal meaning they express. this underlying level of meaning is called connotation and often carries social, cultural, or emotional implications for listeners or readers @xcite . in high-stakes settings such as opinion polls or news reporting, it has been shown that subtle changes in word choices can influence responses or opinions @xcite . for example, the choice of the term 'undocumented workers' vs. 'illegal aliens' to describe immigrants can elicit different levels of prejudice toward that group (m.s., 2010). different connotations can inject multiple layers of meaning into a word, phrase, or passage. this 1 source code is available at https://github.com/shinyemimalef/probing-power-byprompting ## background in this section, we provide an overview of the relevant literature on connotative framing and its formalization, probing plms, and how connotative framing and plms combined can be harnessed to study media bias. ## probing plms for connotative power we propose a probing framework to assess how much connotative information plms acquired during pretraining, including bert @xcite , roberta @xcite and albert @xcite . similar to the masked token prediction objective during plm pre-training, we have plms predict a masked token which in context reveals the power dynamics between an agent and theme in a given sentence. our probe formulation is closely tied to @xcite 's annotation instructions to align the ground-truth data and information elicited from the plms. we directly query the underlying masked language model in plms to compute power connotations associated with predicates. in particular, we formulate our probing template reflecting the definition of the power connotation: probe 1 ag p th. th has mask power than ag. here, p is a placeholder for a predicate (details in §5.2). ag and t h are placeholders for concrete entities in the agent and theme position, which during probing and fine-tuning, are instantiated with common english names (details in §3.1). in the remainder of this paper, we will refer to the first sentence of our template as prelude which introduces the verb and associated entities; and to the second sentence as hypothesis. figure 1 (bottom) shows two instatinated probes. the plms are probed with the context of an instantiated probe with {@xmath0, @xmath1, t @xmath2: p (@xmath3, ag, th), for the predicted probability of two possible values in the mask position, namely the masked target word taking value @xmath4, 'less'}. for example, if p (@xmath5, the plm predicts higher power of theme (th) than agent (ag) of the verb p, (ag≺ power th). to reduce the impact of specific instantiations of entities, we compute the final power connotation score for p as the average score over a large numger of entity combinations in the ag and t h positions: ## fine-tuning plms for connotative power prediction as we will show in §6.1, plms do not encode power connotation naturally (in the zero-shot setting). we therefore aim to instill connotative knowledge into plms through task-specific fine-tuning. probe 1 ag p th. th has mask power than ag. no-ent 1 p. has mask power than. part-sent 1 th has mask power than ag. we use the probes introduced in §3 to construct a connotation prediction task as masked token prediction-based fine-tuning. the input to the model is the instantiated probe with instantiated ag, th, and p and with the target masked word m. the model is trained to predict 'more' or 'less', in correspondence with the gold standard power connotation value for the predicate p, cs p . training then proceeds as normal, backpropagating gradients through the full model, allowing the model to focus better on aspects of the encoding related to power connotation. ## results we first test plms for power connotation knowledge in a zero-shot setting ( § 6.1), before we turn to task-specific fine-tuning ( § 6.2). see appendix a for parameter settings. ## case study we employ our best-performing fine-tuned model roberta to explore subtle power dynamics in us news reports on immigration. in particular, we study the power connotations implied in the descriptions (i.e., actions and events) involving prominent entities in the articles. in doing so, we draw connections between power connotation and emphasis frames ( §7.1) as well as power connotation and issue stance ( §7.2). we first describe our dataset and the entity extraction process, before explaining how we map the predicate-centric probing framework from §3 to an entity-centric measure. dataset we use the media frames corpus @xcite ), which contains 6.7k news articles about immigration, foot_5 manually labeled with one of 15 emphasis frames (including legality, political, economic, . . . ), as well as an additional 42k unlabeled news articles. the articles were sourced from 13 u.s. news outlets published between 1969 and 2017. a subset of the immigration articles includes manual labels of 'stance' which indicates whether the article author is supporting, neutral about, or opposing immigration. 8 entity extraction we identify the most common entities within each article using an offthe-shelf transformer-based semantic role labeling model 9 @xcite which has been previously used to identify key entities in mfc articles @xcite . for each sentence in a document, we collect spans corresponding to the main verbs (predicate), and their first (arg0) and second (arg1) arguments, and then apply a coreference resolution model @xcite to group the arguments into entities. we also use ner and string matching to find all mentions of each argument and consider only entities @xmath6. figure 5 in the appendix b shows the most common entities in the dataset. ## conclusion we introduced a framework to (i) disentangle connotation frames implied by the predicate from its arguments and the sentence structure; and (ii) quantify predicate connotation frames in plms. using the proposed framework, we investigated the capability of pre-trained language models to understand connotative information focusing on power dynamics between involved entities, both in a zero-shot setting, where performance was overall poor, and fine-tuning, which lead to drastic improvements. our framework can be applied to investigate other connotation frames such as agency and sentiment, and their relationships to media bias, although defining the probing template may be challenging due to the subtle nature of connotation frames. finally, in a case study, we showed how our model can be used to detect subtle differences in the implied power dynamics between entities. our findings highlight the potential of our framework as a tool for understanding subtle bias in the media. future work could use our framework to analyze language in various forms of media, e.g. social media posts and tv programming, to identify patterns and trends in language usage for conveying power and other connotations. it is worth investigating whether the improved performance in connotation predictions would enhance the model's reading comprehension @xcite capability at a higher level. naturally, we would like to see that the model can infer which entity is more powerful or wields more authority without the text explicitly stating that fact. previous work has shown that different layers in plms specialize in various meanings @xcite . future work should investigate the extent each layer in a plm contributes to encoding connotation frames. our framework can also be used to study character roles by analyzing the verbs or verb phrases used to describe the characters and the connotations that they carry. for example, a news article about a political leader who is advocating for tougher immigration policies might mention the event of 'cracking down on illegal immigration', evoking a negative connotation and portraying the leader as a 'villain'. a different article on the same entity might include an event of 'helping refugees find safety' assigning the leader the role of 'hero' with a more positive connotative association. both articles frame the immigrants as 'victims'. by analyzing the language used to describe characters and the connotations it carries, it is possible to gain insight into their framing and roles in the broader narratives around an issue.