Generalizability of Media Frames: Corpus creation and analysis across countries (2025)

Abstract: Frames capture aspects of an issue that are emphasized in a debate by interlocutors and can help us understand how political language conveys different perspectives and ultimately shapes people’s opinions. The Media Frame Corpus (MFC) is the most commonly used framework with categories and detailed guidelines for operationalizing frames. It is, however, focused on a few salient U.S. news issues, making it unclear how well these frames can capture news issues in other cultural contexts. To explore this, we introduce \texttt{FrameNews-PT} , a dataset of Brazilian Portuguese news articles covering political and economic news and annotate it within the MFC framework.Through several annotation rounds, we evaluate the extent to which MFC frames generalize to the Brazilian debate issues. We further evaluate how fine-tuned and zero-shot models perform on out-of-domain data.Results show that the 15 MFC frames remain broadly applicable with minor revisions of the guidelines. However, some MFC frames are rarely used, and novel news issues are analyzed using general ‘fallback’ frames. We conclude that cross-cultural frame use requires careful consideration.

## introduction frames are interpretation schemes used to organize reality by isolating and highlighting salient aspects of it @xcite . they combine three levels: cognitive (concerning the mental representations of the world); semantic (concerning the linguistic structures involved); and communicative (their usage and impact on the audience). frames are finding increasing application in nlp as a framework to analyze social media and news coverage. in particular the communicative dimension of frames can be seen as a bridge between belief and action @xcite . following @xcite , frames have four objectives: i) promote a particular definition of a problem; ii) promote an interpretation of it; iii) propose a moral evaluation; and iv) recommend a treatment or a solution. this makes them a valuable tool for identifying, measuring, and quantifying particular worldviews and communication strategies @xcite . frames are primarily a methodological framework, and not a fixed set of categories. however, a widely used set is based on the media frames corpus (mfc), 15 frames (see as in other nlp tasks, however, generalization is an issue: coarse-grained frames risk failing to capture important aspects of a debate, notably ideological bias, while fine-grained frames may not generalize across debates. for example, @xcite demonstrate that in news about migration, the generic frame cultural identity is correlated with liberal texts; introducing two related sub-frames hero: cultural diversity and threat: national cohesion leads to a clear separation by political ideology -liberal in the former and conservative the latter case -but these subframes are presumably irrelevant for other debates. similar questions arise in cross-lingual contexts: the mfc frames were applied by semeval 2023 task 3 to 9 languages @xcite , but the applicability of the frames was not assessed, nor was inter-annotator agreement reported. indeed, full cross-linguistic generalization of frames is far from clear, given previous work on the cross-lingual generalization of semantic inventories @xcite @xcite . for full generalization, frames would show high agreement -their definitions and annotation guidelines would allow each markable in a new language to be assigned to a unique frame -and completeness -for each instance in the new language, there would be a suitable frame. however, we do not have empirical evidence so far. thus, we set out to answer these questions: our study proceeds by creating framenews-pt, a corpus of 300 news articles of 'hard news' @xcite in brazilian portuguese. 1 framenews-pt is based on the news portal recommendation dataset @xcite . it contains brazilian news about, e.g., new regulations, taxes, and finance. we manually annotate the articles with mfc frames using a perspectivist approach @xcite . we carry out several rounds of annotation to a) adapt the mfc guidelines while maintaining their level of detail, b) value the individual perspectives of annotators, and c) examine generalizability and overlap of frames (rq1+2). after corpus creation, we conduct modeling experiments (rq3). we use two transformerbased models fine-tuned on the mfc and 9 chatinstructed models in a zero-shot setting, and analyze how well models perform out of domain on framenews-pt in comparison with the mfc. our results are generally promising: (most) mfc frames are annotated with high agreement in brazilian hard news, with little confusion among them. however, some frames appear rarely, being bound to u.s.-specific issues. also, completeness is not perfect: brazilian issues outside the scope of the original mfc are often annotated with relatively general frames as a fallback (e.g., economic), leading to some loss of information. on the modeling side, we find that zero-shot models perform better at frame prediction on the brazilian data than transferred english classifiers, indicating that the data distributions are sufficiently dissimilar. overall, use of mfc frames in new languages and issues calls for a reflected process. ## related work previous work in nlp has investigated how framing political language influences society and public opinion @xcite , and how it shapes the spread of information @xcite across communication contexts, such as agendasetting campaigns @xcite and news media @xcite . frames can help analyze argumentation, and frame detection can be seen as argument assessment @xcite . given their connection to political ideologies @xcite , frames are also useful for analyzing and diversifying perspectives, e.g. in news recommendation @xcite . the task of frame detection has often been approached through topic modeling with unsupervised methods @xcite @xcite , which, however, tend to be highly corpus-specific. other studies employ supervised classification techniques, including logistic regression @xcite , neural networks @xcite , and fine-tuning of pre-trained models @xcite , all of which require robust gold standard annotations. the media framing corpus (mfc) @xcite , building on the foundational work of @xcite , introduced a novel annotation framework that has become a reference for frame identification (cf. section 3). several studies either use parts of the mfc or applied its guidelines to related datasets, e.g., @xcite conduct a systematic analysis of the 15 frames in new york times articles across 17 years. @xcite are the first to test the generalizability of mfc frames to other contexts. they apply the framework to australian parliamentary speeches on same-sex marriage and migration, and fine-tuned three bert-based classifiers. their findings show a drop in accuracy when the models are applied across different contexts. it is unclear, however, if this is due to the failure of the frames to generalize, or only to a shift in frame distribution. other corpora and labeling schemes have been proposed, though they often focus on specific debate issues. examples include the ballistic missile defense corpus @xcite , the gun violence corpus @xcite , and the migration corpus by @xcite . 2 semeval-2023 task 3 introduced a sub-task on framing detection in online news within a multilingual setting @xcite . the dataset spans multiple topics and languages. it is annotated with an adaptation of the mfc framework @xcite and the proposed models, based on transformer architectures, achieved strong performance. the focus of the study was on modeling, however, not on assessing frame generalizability. ## data the policy frames codebook a key contribution to the study of frames is the policy frames codebook by @xcite , which introduced 15 framing dimensions applied to three debate issues: immigration, tobacco, and samesex marriage. the guidelines for frame annotation in this codebook have been dynamically updated over time @xcite . this work led to the creation of the media frames corpus @xcite , later expanded to include three additional issues: gun control, death penalty, and climate (mfc v4.0). the mfc was explicitly developed for three debate issues and the u.s. context, which is a potential limitation. additionally, as noted by @xcite , it may not distinguish sufficiently between issues and frames by introducing overly broad categories, such as economic. for our study, we adopt the 15 framing dimensions as labeled in @xcite , cf. full list in a topic modeling analysis with bertopic (grootendorst, 2022) on npr and framenews-pt shows that our selection of topics is representative of the whole npr corpus (see appendix a.3). to comply with model input length limitations, we limit our selection to articles shorter than 300 words and remove boilerplate (banners, ads). ## analysis of frame generalization we can now investigate the generalization of the mfc frames to the brazilian context. inter-annotator agreement as figure 2 shows, the agreement score on the primary frame kept improving generally over the four rounds, with the exception of the third round, which contained more complex cases. the discussion was useful in distinguishing difficult frames. after the final revision, the global agreement after discussion on the dataset was α=0.78. it is substantially higher than the one reported on the mfc by @xcite , which never exceeds 0.70. this can be attributed to four main reasons: i) in the mfc, new annotators with no previous knowledge were introduced in each round, potentially lowering the overall score; ii) discussing the guidelines with the same person, as we did, could in the worst case lead to overfitting (we alleviate the effects in modeling by keeping annotations disaggregated); iii) annotators usually fell back to general frames rather than specific ones foot_4 ; iv) they used other to annotate articles about personal stories that did not match other frames in the guidelines, consequently raising the agreement. iaa on single frames we also compute agreement on single frames (figure 3 ). the results confirm highest agreement for other, arguably due to its particular function in our annotations (see section 3). other frames with high scores include external regulation and reputation, suggesting that it is particularly well defined, political, and economic, which is not surprising given that these are the dominant categories in the dataset. on the other hand, the frames with the lowest agreement are capacity and resources and quality of life. (3) ministério diz que novo bolsa família terá r@xmath0 18 billion for children up to 6 years old. crime and punishment and legality, constitutionality and jurisprudence overlap when talking about legal measures to punish someone, while policy prescription and economic occur together when speaking of policies that affect economy. we would argue that this overlap is not due to failure of a frame to generalize. rather, it is a case of a complex situation which is described as having an import on multiple aspects. such multi-class cases are intrinsic to framing. completeness: frame frequency while a low frequency does not imply that a frame does not generalize, high frequencies for other or general frames may indicate lack of completeness. we therefore compare frame frequencies between mfc and framenews-pt (figures 5 and 6 ). some frames, like economic, fairness and equality, and external regulation and reputation appeared more frequently in framenews-pt, suggesting they transfer well to brazilian debate coverage. in contrast, six frames were used fewer than 20 times: health and safety, security and defense, quality of life, capacity and resources, morality, and public opinion. we hypothesize that these frames are not used frequently because they are tied to specific issues in the mfc; for example, capacity and resources is strongly associated to climate change. cultural identity was never used by our annotators, which appears due to its specific use in the u.s. issues migration and gun control. the proportion of other in our corpus is almost the same as in the mfc (13%), although it is used more broadly in our annotations (see section 3). this means that, even though we sample from a much broader range of issues, the 15 frames still suffice to classify the news. however, annotators in our case tended to fall back to more general frames, indicating a lack of completeness. annotator survey to validate our interpretations from above, we asked the annotators to fill a survey about the annotation process, and discussed the results in a meeting. the feedback from the survey confirmed that annotators found our summarized guidelines easier to read and understand than the original one by @xcite . they identified two main limitations. first, some frame descriptions are overly specific to the u.s. context. for example, security and defense included immigration issues at the u.s. border (cf. appendix a.6), a topic not prominent in brazil. second, some frames are not described sufficiently distinctly by the guidelines, leading to overlap. regarding the frequency of frames, the annotators attributed the low usage of certain frames to the topics covered, confirming our hypothesis that some frames are specific to particular issues. however, when asked whether they would add or eliminate any category, they agreed that the same tagset could be used for future tasks, provided that some adjustments are made, making the frames more context-aware and contrastive. the articles tagged as other mainly focused on private issues rather than on societal-level concerns, which is why they did not match any frame. while these could be covered by new frames (e.g., personal finance, advertisement), adding such finegrained categories would be misaligned with the goals of framing analysis for hard news. ## frame prediction on framenews-pt even very good frame generalizability does not imply that frame prediction models transfer well across languages, and at least for some frames we see substantial shifts in frequency (cf. figure 6 ). therefore, a) we run two classifiers trained on the mfc corpus, testing them both on mfc and on framenews-pt; b), we classify framenews-pt in a zero-shot setting with 8 chat-instructed llms ranging from 1b to 12b parameters as well as chatgpt-4o (exact parameter size unknown). all chosen models are multilingual since framenews-pt is in portuguese. ## conclusion this study introduces framenews-pt, the first dataset of brazilian portuguese news with both users' click history (sampled from the npr corpus) and annotation with the 15 media frames corpus frames. it offers a valuable resource for both the nlp and recommender systems communities. by including frame annotations, it also enables new research directions, such as investigating the relationship between frames and topics. our analysis focuses on the generalizability of the mfc frames to the brazilian context. we find generally high agreement among annotators, which we interpret as a promising sign; however, the mfc frames are not complete. in the absence of specific frames, annotators tend to fall back to general ones, with reduced specificity as consequence. this trend is largely due to the original frames and annotation guidelines being developed for the u.s. context on a limited number of policy issues. our modeling experiments show that zero-shot generative models achieve robust results on our dataset. however, the very good in-domain results for supervised mfc models indicate that annotated data is still pivotal to reach strong performance; both model types suffer from shortcomings in frame definition and data creation. we conclude that the 15 mfc frames remain broadly applicable and continue to capture essential dimensions of political discourse, even across countries and languages. to improve generalization, we recommend refining the annotation guidelines to offer clearer distinctions between overlapping categories and incorporating locally relevant examples (e.g., bolsa família instead of medicare). such refinements would enhance the framework's adaptability to diverse cultural contexts and evolving policy debates.