Discovering and Articulating Frames of Communication from Social Media Using Chain-of-Thought Reasoning (2024)

Abstract: Frames of Communication (FoCs) are ubiquitous in social media discourse. They define what counts as a problem, diagnose what is causing the problem, elicit moral judgments and imply remedies for resolving the problem. Most research on automatic frame detection involved the recognition of the problems addressed by frames, but did not consider the articulation of frames. Articulating an FoC involves reasoning with salient problems, their cause and eventual solution. In this paper we present a method for Discovering and Articulating FoCs (DA-FoC) that relies on a combination of Chain-of-Thought prompting of large language models (LLMs) with In-Context Active Curriculum Learning. Very promising evaluation results indicate that 86.72% of the FoCs encoded by communication experts on the same reference dataset were also uncovered by DA-FoC. Moreover, DA-FoC uncovered many new FoCs, which escaped the experts. Interestingly, 55.1% of the known FoCs were judged as being better articulated than the human-written ones, while 93.8% of the new FoCs were judged as having sound rationale and being clearly articulated.

## introduction the way in which we interpret information depends on how the information is framed @xcite @xcite @xcite . for instance, if information about vaccines is framed to build our confidence in them, we can become vaccine enthusiasts. the notion of frame of communication (foc) has emerged from the theory of communication, studied in social sciences. discovering focs is challenging because the focs are not directly expressed in texts, but rather texts evoke them, as shown in figure 1 . framing entails emphasizing specific aspects of a topic within a text, guiding the audience towards a particular understanding. for the text illustrated in figure 1 , which is part of the discourse about covid-19 vaccines on social media, the selected aspects are (1) the calculation people make about the personal costs and benefits of getting vaccinated; and (2) the complacency of getting vaccinated due to low perceived risk of infections. these aspects can be interpreted as problems related to vaccination. the two problems become salient to the foc evoked by the text illustrated in figure 1 . in a widely cited definition, @xcite notes that "to frame is to select some aspects of a perceived reality and make them more salient in a communicating text, in such a way as to promote problem definition, causal interpretation, moral evaluation, and/or treatment recommendation for the item described." this means that, as a minimum, in addition to discovering the salient aspects of an foc, we need to promote a causal interpretation of these aspects by articulating the foc. in the foc evoked by the text illustrated in figure 1 , the problem of calculation is caused by the preference for getting covid-19 and fighting it off. the problem of complacency is caused by the assumption that getting covid-19 is preferable to getting vaccinated. the final articulation of the foc com-bines coherently both these causal interpretations of the problems. we note that the articulation of an foc is expressing the reasons (or causes) of salient problems, but it is not explicitly mentioning the problems, instead it is implying them. therefore the articulation of an foc is a much harder nlp task than the discovery of focs and their salient problems. previous research addressing the problem of foc discovery @xcite @xcite @xcite focused only on the discovery the salient problems implied by focs. this was due to the release of the media frames corpus (mfc) @xcite , which annotates fifteen dimensions of policy frames, addressing such problems as constitutionality and jurisprudence or security and defense. it is important to (1) discover when an foc is evoked by a text; and (2) to be aware of which salient problems 1 are highlighted. however, without articulating the foc, we cannot infer how the text should be interpreted. moreover, without articulating focs, we ignore the many ways in which the same problem is framed in all texts that address it. but, as reported in @xcite @xcite , the communication literature addresses mostly the inductive vs. deductive frame analysis, from which human inference of the articulation of the focs emerges. we believe that the reasoning capabilities of large language models (llms) enable the automatic articulation of focs. this motivated us to design a method for discovering and articulating focs (da-foc). evidently, articulating focs involves reasoning with the problem(s) addressed in texts. moreover, each articulated focs must be relevant, i.e. multiple texts should evoke it @xcite . therefore, discovering and articulating focs must consider that (1) focs may address one or more salient problems; (2) the foc articulation needs to provide a rationale for each salient problem; and (3) the articulated foc should be relevant. these requirements are very burdensome even for communication experts, who typically rely on codebooks emerging from their reasoning and painful inspection of large quantities of texts (kwak et al., 2020b; russell neuman et al., 2014; reese, 2007; matthes 1 the dimensions of the media frames corpus correspond to the problems highlighted by an foc. the notion of frame of communication and media frame are used interchangeably in communication theory @xcite . the recent ability of llms to perform complex reasoning provides an unprecedented opportunity for using them to simultaneously discover and articulate focs. in this paper we explore how chainof-thought (cot) prompting @xcite of llms can be used to reveal not only the problems addressed in texts but also the articulation of the focs. in addition, the cot framework we used for da-foc benefits from in-context active curriculum learning, allowing the llm to learn from its own mistakes. because many focs discovered and articulated in this way may be paraphrasing each other, or they may be specializations of other focs, we also used cot prompting to discover relations between focs. the relations between focs enabled us to select only focs that are relevant. in designing our da-foc method, we focused on social media platforms where millions of users express their opinions and participate in conversations about issues of their interest. in their social media postings (smps), often users select particular aspects, or problems, of an issue, revealing the reasons for their interest in the problem. in doing so, they evoke focs, as shown in figure 1 . in addition to using only smps, which present the advantage of text brevity, we considered only the discovery and articulation of focs regarding covid-19 vaccines. this allowed us to rely on knowledge about salient problems characterizing vaccine hesitancy, reported in @xcite . it also allowed us to make use of the only reference dataset having expert-annotated focs which are articulated. in @xcite 14,180 smps have been expert-annotated with 113 focs. we have enriched this dataset by asking communication experts to also judge which of the problems reported in @xcite were implied in each foc. using this enriched dataset allowed us to train and test da-foc and to make the following contributions: ¡1£ we introduce the first method that does not only discover focs from texts available in smps, but also articulates the focs by using cot prompting of large language models (llms) with in-context active curriculum learning (icacl), a promising new method for prompting llms. ¡2£ we describe the first method of discovering relations between focs, identifying paraphrases, specializations, and contradictions between them. we make available all prompts, annotations, articulated frames, and relations discovered between frames on github foot_0 . ¡3£ a by-product of our method is the identification of all social media postings evoking the same foc, which informs its relevance. ## reference dataset to our knowledge, the only existing dataset of smps annotated with focs is covaxframes, reported in @xcite . this dataset includes focs related to covid-19 vaccination hesitancy. vaccine hesitancy, as reported in @xcite , is characterized by seven factors, or problems, that increase or decrease an individual's likelihood of getting vaccinated. for each of the focs annotated in covaxframes, four researchers have annotated the problems that they address. the problems are listed in step 1: task-specific prompting step 2: initial demonstrations step 3: prompting a large language model llm with chain-of-thought step 4: verify and edit step 5: prompting a large language model llm with test data training data test data editing occurred ??? zero-shot few-shot active learning loop yes no demonstration examples curriculum examples update the demonstrations ## the da-foc method the da-foc method has three distinct phases. in phase a, focs are discovered and articulated using the cot prompting with the in-context active curriculum learning (cot-icacl) framework illustrated in figure 2 . since we noticed that some of the focs articulated in phase a are paraphrases, while some focs were generalizations/ specializations of other focs, and also some focs contradicted each other, we used the same cot-icacl framework in phase b to discover possible relations between focs. because in phases a and b we do not account for foc relevance, in phase c we tackle this necessary property, selecting the final set of focs. ## evaluation results quantitative results: to compare the results of our method with a simple baseline, we considered a methodology that clustered all smps from the test data. clustering was facilitated by creating smp embeddings p * @xmath0. hierarchical agglomerative clustering (hac) was employed from @xcite with a variance gain threshold of 1.1, selected from initial experiments on the training data. for each cluster cl j , the first sentence of the sm p i closest to the centroid of cl j was selected and placed in the set of final focs. obviously, this baseline does not discover any relations between focs. four llms were considered in our evaluations of the da-foc framework: vicuna-13b @xcite , llama-2-70b @xcite , gpt-3.5 @xcite , and gpt-4 (openai, 2023). in phase c we chose t @xmath1, corresponding to each foc needing to be evoked by at least two smps. further discussion surrounding this decision along with ablation results are provided in appendix d. furthermore, active learning loops with a minimum of 50 curriculum examples produced the best results from initial llm experiments. qualitative results: the quality of the final set of focs was evaluated in terms of three properties: (a) the soundness of the rationale provided by the llm when articulating a foc; (b) the clarity of the foc articulation generated by the llm; and (c) the novelty of the final set of focs when compared to the known focs in the reference dataset. two linguists were tasked to judge the soundness, clarity, and novelty of final focs, with n s focs deemed sound, and n c focs deemed clear. with n t final foc proposed by each method, then the quality of reasoning (z) involved in uncovering focs is @xmath2. while metrics z and a capture the soundness and clarity of the final set of focs, we also considered four additional evaluation metrics that account for the novelty of the focs. for each f , which is a clearly articulated foc, an expert linguist was asked to find if f conveys the same information as any f r , representing the focs available from the reference dataset. when f and some f r state the same thing, we consider f to be known, and thus not novel. let n k represent the number of known focs judged in this way, and n f the total number of reference focs. this allows us to define two additional evaluation metrics: (1) the r metric, defined as @xmath3, which models the recall of clearly articulated focs; and (2) r @xmath4. finally, as we desire the focs to be both clearly articulated and fully recalled, we combine the a measure with the r measure into f 1 = 2ar/(a + r). we also are interested in measuring the clarity of the novel focs, and therefore we use the evaluation metric p @xmath5. we also performed an evaluation of the relations between focs discovered by gpt-4 employing cot-icacl, given that this method produced the best results for discovering focs. expert inspection revealed that 96.56% of these relations were correct. more specifically, 99.15% of p-rels were correct, 96.54% of s-rels were correct and 86.30% of c-rels were correct. mistakes are further analyzed in appendix f. ## discussion the results obtained when using cot-icacl with gpt-4 as the llm are not only the best, but they are also impressive across all evaluation metrics. even when using cot-icacl with gpt-3.5 as the llm, our method obtained a substantial improvement over the baseline for all evaluation metrics. but unlike gpt-4, gpt-3.5 does not produce many sound rationales, as revealed by the results of the z metric, showing that its reasoning capabilities are limited when compared to gpt-4 @xcite . gpt-4 enabled the uncovering of many more clearly articulated focs, as captured by the a metric. interestingly, many of the prompting methods were able to have good recall of the known focs, created by experts. but in terms of both clearly articulating focs and revealing all focs, only methods powered by gpt-4 were competitive, given the interpretation of the values of the f 1 metric. furthermore, the values of the p a evaluation results indicate that novel focs, which were not discovered by experts, were well articulated only when the used llm was gpt-4. this makes us conclude that uncovering focs from smps can be performed with high values of soundness, clarity, and novelty when using gpt-4 and can be further improved with cot-icacl. articulation quality: a different way of assessing the clarity of the foc articulation is made possible when focusing only on the final focs (resulting from phase c) which had the same content as some of the reference focs annotated in the reference dataset. for each pair of focs (f k , f r ), where the uncovered f k was judged by a computational linguist to convey the same information as a reference foc f r , the linguist was asked whether the articulation of f k was (a) better, (b) worse, or (c) of the same clarity as f r . the results of these judgments are listed in when using the cot-icacl prompting with gpt-4, we found that the 586 p-rels between focs discovered allowed us to filter out 1,216 of the uncovered focs, as they were paraphrasing other focs. in addition, the s-rels allowed us to generate 130 foc taxonomies, spanned by s-rels. these taxonomies contained on average 6 focs. the largest taxonomy contained 49 focs, with a depth of 7. sometimes, in a foc taxonomy, there were focs specialized as many as 13 times. the taxonomies will enable further research on the ideal specialization of an foc articulation. we also found that the final set of focs contained 43 pairs of contradicting focs, demonstrating that opposing viewpoints were common. an interactive website enabling an exploration of the discovered focs, foc relations, and foc taxonomies has been made available foot_1 . figure 4 illustrates how this interactive website operates. each node represents one of the final focs discovered when using the cot-icacl promting of gpt4, with the colors corresponding to the problems identified by cot reasoning. edges in the graph represent specializing and contradicting relations, since all paraphrases have been eliminated. zooming in on the full graph enables an exploration of the various automatically constructed foc taxonomies, and hovering over each node provides the articulated foc along with the identified problems and the number of smps evoking the foc. hovering over the edges also provides the rationale justifying the relation spanning the pair of focs. ## related work initial large-scale research on frame identification from social media has generally relied on unsupervised approaches @xcite @xcite which revealed interesting framing patterns, highlighted by lexical terms, but did neither articulate any foc nor discover any problems that focs address. classifiers aiming to identify frame-invoking language were reported in @xcite , but these classifiers did not identify the problems addressed by focs. the assumption that frames can be associated with certain stock phrases was chal-lenged in @xcite , showing that frames can also be associated with certain topics. a growing body of research using supervised nlp methods uses the media frames corpus (mfc) @xcite . these methods detect frame salient problems with techniques including logistic regression @xcite , recurrent neural networks @xcite , lexicon induction @xcite , and fine-tuning pretrained language models @xcite . furthermore, subcategories of the policy frame dimensions annotated in mfc were extracted with a weakly-supervised approach @xcite . the only prior work that considered the analysis of frames in social media was reported in @xcite , where immigration policy problems were identified in smps with multi-label classification methods, relying on roberta @xcite . all these prior methods do not articulate focs, they only discover them. we believe that the release of the reference dataset used in our work, which annotates both focs and the problems they address, will facilitate new research in the difficult problem of discovering and articulating focs. finally, none of the previous methods have considered the need to learn to automatically provide a rationale for the discovered focs or for their salient problem(s), which our da-foc method enables by using chain-of-thought prompting of llms with in-context active curriculum learning. ## conclusion this paper presents a new method capable to discover and articulate frames of communication from social media. by combining chain-of-thought prompting of llms with in-context active curriculum learning, both previously known and especially new frames were revealed. extensive evaluations show that when using gpt-4 with cot-icacl, 86.73% of the frames identified by experts were re-discovered on the same dataset while also uncovering many new frames, which are both clearly articulated and sound. the rationales generated by gpt-4 with cot-icacl help us to make sense of these uncovered focs, providing additional insights for understanding why certain problems are discussed on social media. the relations between frames help us discover when some frames specialize others and when some frames contradict others.