A Study on Scaling Up Multilingual News Framing Analysis (2024)

Abstract: Media framing is the study of strategically selecting and presenting specific aspects of political issues to shape public opinion. Despite its relevance to almost all societies around the world, research has been limited due to the lack of available datasets and other resources. This study explores the possibility of dataset creation through crowdsourcing, utilizing non-expert annotators to develop training corpora. We first extend framing analysis beyond English news to a multilingual context (12 typologically diverse languages) through automatic translation. We also present a novel benchmark in Bengali and Portuguese on the immigration and same-sex marriage domains.Additionally, we show that a system trained on our crowd-sourced dataset, combined with other existing ones, leads to a 5.32 percentage point increase from the baseline, showing that crowdsourcing is a viable option. Last, we study the performance of large language models (LLMs) for this task, finding that task-specific fine-tuning is a better approach than employing bigger non-specialized models.

## introduction news framing refers to the power of the news media to define and interpret events, issues, and policies by emphasizing certain aspects while downplaying or excluding others. according to @xcite , it can "make a piece of information more noticeable, meaningful, or memorable to audiences". it plays a crucial role in influencing how people interpret and react to information presented in news articles. the language used in news media can shape public opinion and reveal biases and agendas, which can ultimately shape the way people understand and react to current events. 1 code and dataset available here: https://github.com/syedasabrina/ scaling-up-multilingual-framing-analysis.git traditionally, framing analysis has relied on manual annotation by linguists, social studies experts, and trained annotators, lacking the potential of ai-driven systems leading to a rather limited explorations of automating framing analysis. moreover, existing studies have been restricted primarily to english-only data, leaving a gap in research concerning multilingual and low-resource contexts. our work focuses on employing nlp techniques for the framing analysis task to automate the analysis process, extract insights from large datasets efficiently, and identify patterns in the language used in news media. to address these challenges, @xcite introduced a codebook, policy frames codebook, based on which the media frames corpus (mfc; @xcite was created. this dataset is comprised broad categories of common policy frames and annotations of us news articles. however, the availability of such datasets in languages beyond english remains limited. getting a higher volume of higher quality data (such as, mfc) is time and resource intensive. hence, we study the alternative of gathering a high volume of comparatively lower quality but easyto-collect data. we achieve this through crowdsourcing and automatic translation techniques. we also examine the combination of lower and higher quality data. in this study, we first introduce a new crowdsourced dataset: student-sourced noisy frames corpus (snfc). we have achieved time and cost efficiency by involving a large number of semitrained annotators for the data collection and annotation process of the corpus. snfc covers immigration and same-sex marriage domains and includes novel benchmark test sets in bengali and portuguese, offering new perspectives in these languages. additionally, we automatically expand multilinguality to the task by translating the mfc and snfc to 12 more languages. we show that a neural classifier trained on the combination of both mfc and snfc yields significant performance improvements, both in english as well as in a multilingual setting. finally, we explore generative large language models, such as llama @xcite , to study their efficacy for this task. our findings show that neural models trained on snfc can reach the performance levels of those trained on high quality data (i.e., mfc). going further, we find that the combination of expert and non-expert annotated data (i.e. masnfc+mfc) outperforms just mfc, which provides a path towards expanding coverage without the need for expensive expert annotations. ## related work framing analysis provides valuable insights into different perspectives on news topics across various countries and languages. however, there is a notable lack of research and annotated corpora for framing analysis in languages other than english. this limitation hinders our understanding of media framing in different parts of the world and other societies' opinion regarding specific issues. to address this gap, a multilingual approach is essential in analyzing media framing across diverse linguistic and cultural contexts. @xcite provide a comprehensive survey of the framing analysis task, focusing specifically on studies in english datasets exploring various approaches and techniques employed in framing analysis. two prominent datasets used for framing analysis are the media frames corpus (mfc; @xcite and the gun violence frames corpus (gvfc; @xcite . the mfc, annotated according to the guidelines provided in the codebook of @xcite , covers 6 different political issues including immigration, same-sex marriage, and gun violence, among others. it includes both article headlines and news texts, providing a broader and more comprehensive dataset. on the other hand, the gvfc focuses solely on the topic of gun violence, with 10 manually annotated frames defined in a different codebook, and it only includes article headlines. akyürek et al. (2020) extended the gvfc by curating headlines in german, turkish, and arabic following the same process as the original dataset from the respective news websites, specifically targeting keywords related to gun violence and mass shootings. the frames used in the multilingual datasets remained consistent with those in the gvfc, and is the one of the few multilingual sources for this task. additionally, the australian parliamentary speeches (aps) dataset @xcite offers another perspective on framing analysis, as it consists of transcripts speeches related to same-sex marriage bills presented in the australian parliament. although the aps dataset focuses on data from a country other than the united states, it is still limited to english language texts, which narrows the scope of the framing analysis task. the mfc has served as a valuable resource in various framing-related studies. for example, it was used to develop a semi-supervised model by extracting a russian lexicon from their russian test corpora which consists of news articles sourced from reputable russian newspapers @xcite . in a different vein, @xcite used it to benchmark sentence-level classification tasks, employing lstm, bilstm, and gru-based systems. considering the significant contributions of this corpus to the field, we have incorporated it into our system for training and evaluation purposes, alongside our snfc dataset. several studies have employed various techniques such as topic modeling @xcite @xcite , cluster analysis @xcite , and neural networks @xcite @xcite to construct systems for framing analysis. these investigations have consistently demonstrated that leveraging state-of-the-art pre-trained models based on transformers @xcite @xcite is a highly effective approach, yielding significantly improved results compared to other techniques. in our study, we follow the state of the art and build models similar to those employed by @xcite and @xcite . we also investigated crowdsourcing methods which, as defined by @xcite , is an online, distributed problem-solving and production model that leverages the collective intelligence of online communities for specific goals. this technique aims to tap into the global talent pool, accelerating innovation and problem-solving across various domains. @xcite provide a comprehensive literature review, identifying numerous crowdsourcing methods, which emphasizes the difficulty of generalizing these methods due to their diversity and application-specific nature. however, the widespread use of these methods demonstrates versatility and adaptability of different crowdsourcing methods. @xcite suggest that future research should focus on standardizing crowdsourcing processes to enhance efficiency and effectiveness. this indicates an increasing realization of the necessity to codify crowdsourcing approaches, notwithstanding their inherent variability. ## dataset creation in this section, we present our methodology for curating snfc training dataset through crowdsourcing ( §3) and outline the process of extending the dataset to incorporate multilinguality ( §3). lastly, we introduce our innovative portuguese and bengali benchmarks, highlighting their significance in the context of this study ( §3). ## framing analysis system and results experimental setup we approach the task as a multilabel classification problem @xcite , leveraging the pretrained roberta @xcite language model, similar to the sota approach employed by @xcite . for all models we set the maximum sequence length to 256, with a batch size of 16,and train using a learning rate of 10 -5 . to expand to more languages, we employ the multilingual xlm-roberta model @xcite . throughout all experiments, we use the base model size. 5 we first report results with models exclusively trained on mfc, and snfc datasets, as well as their concatenation. to investigate a more datascarce scenario, we also compiled a smaller sample consisting of about 10% of the original mfc, 5 appendix 8 and 9 also provides results with the bert and mbert @xcite named mfc10, ensuring all 15 target labels are included. beyond the single-dataset baselines, we combine the expert-annotated mfc and mfc10 with our crowd-sourced snfc. to further study the effect of the size of the snfc, we have experimented with snfc50, a randomized halved subset of the original snfc that is more closer to the mfc in size. ## generative language models llms like gpt-4 (openai, 2023), falcon @xcite , and llama @xcite , are trained on vast amounts of text and have shown immense promise in a variety of nlp tasks. their broad knowledge base qualifies them as potential tools for framing analysis. in this study, we have also explored three of these models, particularly the open-sourced ones: mistral, llama-2, and falcon. ## conclusion in conclusion, our study emphasizes the importance of data quality and language diversity in multilingual framing analysis. combining the media frames corpus (mfc) with the student-sourced noisy frames corpus (snfc) yields significant improvements, highlighting the value of larger datasets despite the annotation quality potentially being lower. however, lower accuracy in multilingual experiments indicates the need for accurate translations and culturally diverse training data to improve multilingual framing analysis. last, the sub-par performance of llms showcases a future research direction towards task-specific finetuning of the llms.