O pen F raming: Open-sourced Tool for Computational Framing Analysis of Multilingual Data (2021)

Abstract: When journalists cover a news story, they can cover the story from multiple angles or perspectives. These perspectives are called “frames,” and usage of one frame or another may influence public perception and opinion of the issue at hand. We develop a web-based system for analyzing frames in multilingual text documents. We propose and guide users through a five-step end-to-end computational framing analysis framework grounded in media framing theory in communication research. Users can use the framework to analyze multilingual text data, starting from the exploration of frames in user’s corpora and through review of previous framing literature (step 1-3) to frame classification (step 4) and prediction (step 5). The framework combines unsupervised and supervised machine learning and leverages a state-of-the-art (SoTA) multilingual language model, which can significantly enhance frame prediction performance while requiring a considerably small sample of manual annotations. Through the interactive website, anyone can perform the proposed computational framing analysis, making advanced computational analysis available to researchers without a programming background and bridging the digital divide within the communication research discipline in particular and the academic community in general. The system is available online at http://www.openframing.org , via an API http://www.openframing.org:5000/docs/ , or through our GitHub page https://github.com/vibss2397/openFraming .

## introduction we live in a world saturated with media. any major public issue, such as the ongoing covid-19 pandemic and the black lives matter protests, attracts * contributed equally † while at summer research in boston university ‡ corresponding author 1 best viewed with google chrome browser tremendous attention from hundreds of thousands of news media outlets -traditional and emerging -around the world. the reporting angles on a single issue are often varied across different media outlets. in covering covid-19, for example, some media outlets focus on government response and actions while others emphasize the economic consequences. social science scholars call this process media framing. to define, or to frame, is "to select some aspects of a perceived reality and make them more salient in a communicating text" @xcite . when used in news articles, frames can strongly impact public perception of the topics reported and lead to different assessments by readers @xcite , or even reinforce stereotypes and project explicit and implicit social and racial biases @xcite . frame discovery in media text has been traditionally accomplished using methods such as quantitative content analysis @xcite , which is a manual method widely used by social scientists. however, in the emerging media environment, the sheer volume and velocity with which content is generated makes manual labeling increasingly intractable. to overcome this "big data" challenge, our cross-disciplinary team, which consists of computer science and communication researchers have employed methods based on sota machine learning (ml) techniques to detect frames automatically and robustly @xcite . however, these sota ml models are not readily accessible to social sciences scholars who typically do not have machine learning and programming training. the current ecosystem around "big data" creates new digital divides between the big data rich and the big data poor @xcite . among other barriers, the limited access to computational resources and skillsets prevents many communication scholars from taking advantage of a large number of unprecedented ml models of the day and hampers their ability to glean valuable insights from unprecedentedly large media datasets since framing, especially at scale, defines how news media coverage shapes mass opinion. our goal is to make computational framing analysis accessible to researchers from a diverse array of disciplines. we present openframing ( www.openframing.org ), a free and user-friendly web-based system that allows researchers to conduct computational framing analysis without having to write and debug complex code. there does, of course, exist click-and-play commercial software, but these tools are often costly and pose issues for researchers due to a lack of transparency into their inner computational mechanisms. this black box problem is present in various applications of data science techniques in communication research @xcite , thus hindering the open science movement in the field @xcite . in contrast, our system is based on sota framing research and our code is publicly available under mit license. specifically, we propose a five-step analytical framework (figure 1 ) allowing users to identify frames in large-scale media text by leveraging sota computational frame analysis research techniques. our work is advantageous in at least five aspects. first, the framework is grounded in media framing theory, one of the most established theories in communication research @xcite . second, we provide a web-based, user-friendly graphic interface where researchers with little or no computational background can perform advanced data analysis through a click-and-play approach. third, all algorithms used are open to users and the benefits and limitations of the algorithms are explained at each step. fourth, given the increasing importance of understanding information flow at a global scale, our tool can be used to analyze media content in 23 languages based on a sota multilingual language model @xcite . lastly, with the support of a research grant, we make our tool entirely free to the academic community. we will start with a review of the theoretical and methodological backgrounds based on which our system is developed. we will then detail the five-step framing analysis facilitated by the system. the importance of bridging the digital divide in the field of computational communication research will also be discussed. ## related work to frame is "to select some aspects of a perceived reality and make them more salient in a communicating text" @xcite . like any type of communication, news involves framing. the ideology of a society, ownership of a news organization, media routine, as well as individual media worker's preferences all play some role, consciously or not, in shaping the news content @xcite . @xcite defines media frames as "organizing principles that are socially shared and persistent over time, that work symbolically to meaningfully structure the social world". in other words, one can frame an issue in multiple ways, but a frame must be shared by the target audience on some level for it to be communicable and effective. accordingly, news frame analysis should focus on frames that are "persistent over time". these include generic frames that appear across issues, time, and space, such as human interest, conflict, attribution of responsibility, and economic consequences @xcite @xcite . for any particular issue, journalists also apply issue-specific frames e.g., reporters often use peace-and war-oriented frames to help their audience understand the complexity of wars @xcite . empirically, communication researchers have developed a variety of approaches to analyze frames. there are in general three computational approaches: 1) lexical-based, 2) unsupervised ml, and 3) supervised ml. the lexical-based approach relies on predefined lists of words, known as lexicons or dictionaries, with each word associated with a certain frame @xcite . for example, @xcite develops keywords to search for frames in the news coverage of immigration. we, however, contend that the lexical-based approach is not ideal for news frame analysis. unlike the topic-like frames in @xcite , many enduring media frames (e.g., conflict, human interest) are abstract and involve complex meanings, which cannot be easily captured by a list of words and terms. the other two approaches are based on ml models that learn from data. while unsupervised ml models discover patterns of frames from unlabeled data, supervised ml is done by training a model on a sample of documents that are labeled with the "correct" frame. in communication research, the "correct frame" often refers to labels provided by human coders through quantitative content analysis @xcite . several existing news frame studies used an unsupervised ml approach. the latent dirichlet allocation (lda) topic modeling @xcite is a popular example (see @xcite for a systematic review). in analyzing news content, the text is observed as a set of latent "topics" and these topics are distributed over words in a probabilistic order. the output of the lda topic modeling is a "topic matrix" with a list of keywords representing each topic. researchers will have to review the top keywords and decide on a label to represent the meaning of each topic. some studies approach media framing by analyzing mainly themes or topics. this approach is problematic because, again, news frame analysis should identify patterns that endure over time, which is different from thematic or topical analyses that describe themes or topics as instances reported in certain stories @xcite . given this, the lda approach is most useful for exploratory analysis. although the lda-generated topics are not necessarily equivalent to frames, the information can be used to obtain initial ideas about the data and infer potential frames for the next step of supervised frame analysis @xcite . supervised ml also has become increasingly common in communication research @xcite . our recent studies use bert language model @xcite and fine-tune it for the task of identifying frames in the news coverage about us gun violence which demonstrate a high level of accuracy for multilingual frame detection @xcite with a relatively small amount of data: 1.3k english frame-labeled news headlines. based on the review of the literature, we propose a five-step end-to-end multilingual framing analysis framework that combines unsupervised and supervised ml. unsupervised ml can help develop a holistic picture of large-scale text corpora, but is not sufficient for a news frame analysis. adding a supervised approach to the research framework is essential because the goal is to identify enduring frames-generic and issue-specific frames to contribute to the media framing literature. furthermore, since developing multilingual ground truth labels through manual content analysis is labor-intensive and time-consuming, to address this challenge in part, our proposed framework incorporates a sota multilingual language model such as bert that allows transfer learning from pretrained models to the task at hand. as a result, it would require relatively fewer labeled documents to fine-tune, while still achieving robust prediction performance. ## system description: five-step multilingual framing analysis our analytical framework, available publicly via www.openframing.org , involves five steps that researchers can take to conduct a end-to-end computational framing analysis of multilingual text corpora (figure 1 ). to help demonstrate the research procedure, we will accompany the description of our system by a case study that exam-ines frames in the u.s. news coverage of the u.s. gun violence issue-also provided as a demo at http://www.openframing.org/demos.html . before using the system, researchers should first collect data related to the issue under consideration. as for the case study, we collect news headlines using the keyword combination (gun or firearm or nra or "2nd amendment" or "second amendment" or ar15 or "assault weapon" or rifle or "brady act" or "brady bill" or shooting) and collected a total of 42,917 u.s. english news articles from 2018 using brandwatch consumer research 2 . ## discussion and conclusion as we have argued, applying sota computational methods to framing analysis can make a significant methodological contribution to the field. however, the implementation involves at least two challenges: the lack of computational resources and skills. deep learning models such as xlm-roberta, are extremely large ml models with millions of parameters to train. even fine-tuning them requires computers with gpu compute capability, which is expensive and not widely available. some cloud services such as google colab provide free but limited access to gpus. the intensive computational requirements for running large ml models and the unequal access to these computing resources among researchers contribute to the digital and compute divide @xcite . for communication researchers, the divide is exacerbated due to the shortage of computational research skills. implementing and fine-tuning topic modeling or deep learning models such as the ones introduced here requires the understanding and a considerable degree of comfortability with using a deep learning programming framework such as pytorch, machine learning libraries such as scikitlearn for the training and evaluation setup and analysis, nlp libraries such as nltk to clean and preprocess the text, as well as python programming that is required for using these frameworks and libraries. users with little experience in computer science or programming would find it challenging to run the computational analysis on their own. with the support of a cross-disciplinary team, our system aims to make the computational fram-ing analysis accessible to researchers with no or little prior experience in computer science and programming. through a click-and-play web-based system, the users can follow the guidance on the website and run the advanced computational analysis step by step. users with different levels of expertise would know where to start and how to interact with the system. also unlike many of the similar applications in the market, our system prioritizes transparency in its data processing and algorithms. the tool is entirely open-sourced and users will have access to the raw code on our github page.