Capturing Topic Framing via Masked Language Modeling (2022)

Abstract: Differential framing of issues can lead to divergent world views on important issues. This is especially true in domains where the information presented can reach a large audience, such as traditional and social media. Scalable and reliable measurement of such differential framing is an important first step in addressing them. In this work, based on the intuition that framing affects the tone and word choices in written language, we propose a framework for modeling the differential framing of issues through masked token prediction via large-scale fine-tuned language models (LMs). Specifically, we explore three key factors for our framework: 1) prompt generation methods for the masked token prediction; 2) methods for normalizing the output of fine-tuned LMs; 3) robustness to the choice of pre-trained LMs used for fine-tuning. Through experiments on a dataset of articles from traditional media outlets covering five diverse and politically polarized topics, we show that our framework can capture differential framing of these topics with high reliability.

## introduction issue framing refers to the ways in which information organizations (e.g. media companies, government institutions, etc.) package and present information @xcite . the framing of the information is of great importance because it influences readers' understanding of the content @xcite . issue framing can be expressed explicitly with evaluative language (e.g., "good" or "bad") or implicitly (e.g., "undocumented" vs "illegal" immigrants) @xcite . measuring the ways in which topics are framed differently (what we term "differential framing") is helpful in identifying the various stances of these information organizations. while much research has been done on qualitative analysis and the effects of issue framing (e.g., see @xcite @xcite ). quantitative methods have been used for this task, though these methods usually utilize hand-crafted features such as the tone or the choice of phrases when referring to the same events or entities (e.g., undocumented immigrant vs. illegal alien). these methods either rely fully on human annotation @xcite @xcite @xcite @xcite -which makes them not scalable and prone to human bias-or utilize (semi-) automatic methods by representing the problem as a standard supervised learning task @xcite @xcite , relying on context-specific hand-crafted features. to address the challenges of scale and topic-/context-specificity, inspired by prior works on prompt-based knowledge extraction @xcite @xcite @xcite which shows that knowledge learned by lms can be extracted through the masked language modeling (mlm) task, we propose a novel method that utilizes lms to measure differential framing in any context or domain. specifically, we capture the tone and word preference of different sources by fine-tuning pretrained lms on corpora from them. through the mlm objective, we then use these fine-tuned lms to predict the choice of words for each source for different contexts, such as different issues or events. in other words, prompting these models with sentences such as "the greatest threat to the immigration system is ___", they will output the word most likely to complete the sentence given the associations learned from their fine-tuning texts. comparing outputs from models fine-tuned on different sources can illuminate key differences in their framing of issues. we explore three key factors in the development of our framework: 1) how to generate the prompts; 2) how to normalize the output of fine-tuned lms; our proposed framework for estimating differential framing is highly scalable and generalizable in that it can be applied to other contexts and domains with minor adjustments. we make our code and data publicly available on github foot_0 . ## dataset construction our dataset is comprised of articles in english collected from media cloud foot_1 , whose topic-labeling has proved reliable @xcite . specifically, we sample news articles under five diverse topics: "climate change", "corporate tax", "drug policy", "gay marriage" and "the affordable care act". the articles are from 10 us news outlets: breitbart news network, cbs news, cnn, fox news, huffpost, new york times, npr, usa today, wall street journal, and washington post. we divide the dataset into training and development (dev) sets with a 90/10 split. due to the length limit of lms, we break the news articles down to their original paragraphs; paragraphs exceeding 256 words are split greedily at the sentence level to ensure the completeness of sentences. we then label the paragraphs with their respective media outlets. in the final dataset, there are 531,398 instances. the statistics of these datasets are listed in appendix a (table a1 ). ## methodology as shown in figure 1 , with the fine-tuned lms for each source, our framework is comprised of three steps: (1) we generate prompts for the mlm task; (2) with the given prompts, we then leverage the fine-tuned lms to create representations of the sources via prompt-based mask token prediction; and (3) we utilize the generated representations to measure differential framing by the sources. we experiment with three different fine-tuned models (using the mlm task), "roberta-base" @xcite , "bert-base-cased" and "bert-baseuncased" @xcite . ## experiments 3 we explore and evaluate our framework in the context of differential framing of topics by different media outlets. appendix a describes the construction and details of the media outlets data used. we evaluate the performance of our framework through human evaluations and three survey-based datasets. appendix d.1 describes the collection of our human evaluation dataset. the construction of the other three survey-based datasets is described in appendix e.1. we compare the similarity rankings of the outlets predicted by our framework with the ones generated using the four datasets described above. specifically, we measure the relative rankings of the outlets based on our ground truth datasets using a similar procedure as described in section 3.3: for each outlet, we calculate the absolute distance to the other outlets (based the ground truth data) and create a ranked list. we then calculate the similarity between the predicted and ground truth similarity rankings via kendall rank correlation coefficients (kendall's τ ). kendall's τ is calculated for each media outlet, and the mean of all the τ 's is used as the evaluation score. finally, we compare the performance of our framework against four baselines. for our framework, we use the combination of three factors achieving the best performance: "interrogative", "one sentence", and domain-normalization. two classic baselines are based on latent dirichlet allocation (lda) and term frequency-inverse document frequency (tf-idf) features. the other two baselines utilize language models ("roberta-base", "bert-base-cased", or "bertbase-uncased"), with one using the classification objective (lm-c), a variant of previous work @xcite , and the other using the mlm objective (lm-m). the lda and tf-idf (up to trigrams) are trained with all media outlet instances and utilized to embed the instances. following previous work, lm-c is fine-tuned with the task of predicting the source of each instance and leveraged to embed all instances in the dev sets. for lm-m, we use our mlm fine-tuned language models to embed the instances in the dev sets. in all cases, we regard the mean embedding of all instances for one media outlet as its corresponding media embedding. for all the baselines with the media outlet embeddings, we calculate the cosine distances between pairs of media outlets in a manner similar to what was described in section 3.3, i.e., for each outlet, we use the distance to other outlets to create a ranked list and then use kendall's τ to measure agreement with the ground truth rankings. appendix c.2 describes the details of the hyperparameters of the baselines. ## error analysis to better understand what our model is capturing, we rank the differential framing for each masked instance and calculate the kendall's τ based on the instance-level rankings. we observe that all distributions are right-skewed distributions with a roberta bert-un we further analyze the instances with the worst and best performance of each topic and ground truth dataset. we observe that the instances with the worst performances (i.e., small τ ), have masked tokens that are easy to predict by all models no matter which articles they were fine-tuned on, by either being a stop word or a named-entity (such as "nancy pelosi"). for example, in the following sentence: "... age-based tax credits in the bill replacing those in ___ affordable care act are too small", the masked token is "the", and all models correctly predict it with the probability above 0.99. we also analyze the instances which achieve the best performance. as hypothesized, in many of the instances the difference in the prediction can be explained by the difference in opinion of the outlets. for example, in the sentence "president donald trump's vague platitudes about beautiful and very ___ health care for everybody...", the masked token is special. for the right-wing media outlets (such as fox and breitbart), the models tend to predict positive tokens (such as "effective" and "affordable"), and for the left-wing media outlets (such as cnn and new york times), the models will predict both positive ("affordable") and negative ("expensive") tokens. ## conclusion and future work we present a framework for capturing differential framing of issues using the mlm objective through masked prompting of fine-tuned large-scale lms. we explore three factors: prompt generation methods, probability normalization, and robustness to different lms. we show that manually designed prompts can achieve slightly better performance than the automatically generated prompts at the cost of volatility. we also show that the normalization of outputs from the fine-tuned lms can greatly boost the performance of our framework. finally, we show that our framework is robust to the choice of lms. we evaluate our framework on differential framing of issues by media outlets, using human evaluations and three survey-based datasets, showing that our framework's predicted differences in the ways in which outlets frame issues mainly agree with these ground truth datasets. future work could investigate the applicability of our framework to other domains.