Multi-Label and Multilingual News Framing Analysis (2020)

Abstract: News framing refers to the practice in which aspects of specific issues are highlighted in the news to promote a particular interpretation. In NLP, although recent works have studied framing in English news, few have studied how the analysis can be extended to other languages and in a multi-label setting. In this work, we explore multilingual transfer learning to detect multiple frames from just the news headline in a genuinely low-resource context where there are few/no frame annotations in the target language. We propose a novel method that can leverage elementary resources consisting of a dictionary and few annotations to detect frames in the target language. Our method performs comparably or better than translating the entire target language headline to the source language for which we have annotated data. This work opens up an exciting new capability of scaling up frame analysis to many languages, even those without existing translation technologies. Lastly, we apply our method to detect frames on the issue of U.S. gun violence in multiple languages and obtain exciting insights on the relationship between different frames of the same problem across different countries with different languages.

## introduction the worldwide image of the united states has dropped precipitously during the past few years @xcite . among other factors, the increasing number of gun violence incidents appears to affect the u.s. reputation abroad. whenever a fatal mass shooting happens, it often attracts significant international news attention. while the domestic u.s. news media often links gun violence to individual shooters' mental illness @xcite , foreign media may attribute it to u.s. gun policy and its gun culture e.g., @xcite . this phenomenon is known as media framing, which is the process of selecting "some aspects of a perceived reality and @xcite them more salient in a communicating text, in such a way as to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment recommendation for the item" @xcite . when foreign media frame the gun violence issue in a way to depict the u.s. as an unsafe and undesired place, it erodes the country's "soft power" @xcite . evaluating how different countries frame the u.s. gun violence issue will enrich our understanding of the u.s. soft power in particular and international relations in general. in this work, we develop a multilingual approach to automatically detect frames in news coverage of different languages, thus facilitating the analysis of how different countries with different languages frame a particular issue. aside from enabling this understanding of foreign public opinion regarding a certain issue or nation, a multilingual approach is essential in media framing analysis, as it is also an understudied problem in many parts of the world. given frame-annotated news headlines of a particular topic in a source language (e.g., english), our approach uses word-to-word translation to translate keywords that are indicative of the frames in these headlines to a target language. then, we fine-tune a state-of-the-art multilingual language model multibert @xcite to detect frames on these "code-switched" headlines, combined with a few annotated headlines from the target language. the translated keywords and a fewshot examples act as anchors to adapt multibert to detect frames in the target language. this approach performs comparably if not better than a model trained on the source language and tested on headlines that are translated from the target language to the source. since our approach requires only simple resources -a dictionary and a few (≤40) annotated examples in the target language -it is handy for many languages. moreover, considering the significant improvement gained over the zero-shot transfer, the proposed approach is much more reliable for languages without existing translation technologies or expert annotations. due to the subtle nature of framing, it is not uncommon for one news article to involve more than one message. communication researchers have suggested that the association of different constructs, such as issues and frames in the news, will influence how the audience associate these elements, thus determining how they perceive the world @xcite . the network agenda setting model suggests that examining the interrelationships between media elements enables researchers to measure media effects in a more nuanced manner. note that some frames appear more often than others. in this work, we formulate our frame detection model to allow for multilabel frame detection while also addressing the imbalance in the frame distribution by adapting focal loss @xcite into our multi-label setting. our multi-label approach allows for the examination of frame co-occurring, or "associative frames" @xcite , across the news articles. overall, the contribution of this work are manifold: today's international politics not only revolve around military and economic influence but also largely depend on a country's soft power @xcite . for each nation, constructing a positive country image to the outside world is crucial to ensure its international competitiveness in this global information society @xcite . in this light, more and more governments have realized the importance of public diplomacy, making great efforts to promote their countries' values and perspectives to foreign publics @xcite . however, these efforts are not always successful. editors of international news media serve as the gatekeepers to decisions which may lead to the framing of a given country contrary to how its government intends. in reporting news about a foreign country, news editors and reporters make conscious or unconscious choices to emphasize specific issues, or emphasize certain aspects of a given topic, which may alter the country's image in the minds of their audience. a multilingual approach is essential to analyze media framing in different parts of the world, which will shed light on foreign public opinion regarding a particular nation. communication researchers often rely on manual content analysis to examine media framing in news outlets of different languages (h. de vreese, 2001). one critique for this type of study is that researchers tend to decide countries for review based on languages spoken in the research team rather than theoretical rationales. this language constraint becomes a more significant challenge in this increasingly globalized media landscape; capturing a holistic picture of international communication would require the analysis of news coverage in a larger number of languages. arguably, an automatic, multilingual approach of framing analysis would greatly benefit the international communication research community. in nlp, language models have been effectively fine-tuned or used in downstream tasks such as text classification @xcite @xcite . further, the introduction of deep contextual language embedding such @xcite , which uses bi-directional lstms and bert (bi-directional encoder representations from transformers) @xcite , has been another milestone in this line of work. bert is currently one of the state-of-the-art models in language modeling. news framing was first brought to the attention of the computational linguistics community by the media frames corpus @xcite , which addresses three issues: immigration, tobacco, and same-sex marriage. @xcite analyzes the framing of the u.s. and agenda-setting in russian news. our work is similar to @xcite in terms of using npmi to find essential words. furthermore, our work advances previous research by leveraging a multilingual language model, facilitating transfer learning in news framing, and relying on parsimonious resources, that is, 50,000 lexical translations vs. ~350 in our case. the current state-of-the-art model @xcite for frame detection fine-tunes bert on frame-annotated english news headlines with the standard multiclass focal loss objective @xcite . their approach predicts only a single frame, which is insufficient given the multifaceted nature of news framing in which multiple frames often co-occur in the same headline. indeed, more than a quarter of the gun violence frame corpus (gvfc) has more than one frame @xcite . in this work, we fine-tune multibert to detect frames in multiple languages' headlines with our multi-label focal loss. our approach can predict (and be evaluated on) multiple frames for each headline, which is a more complex task while being comparable to their work in terms of the average f1 performance. similar to their work, we detect frames on news headlines as they provide the most direct clue to the potential influence of the news coverage. ## dataset creation gvfc is a dataset of news articles from 21 major u.s. news organizations related to u.s. gun violence that contains news headlines and their domain-expert frame annotations @xcite . we extend gvfc to include headlines in other languages by following their process of curating gvfc. we first drew our sample of news articles from german-, turkish-, and arabic-speaking news websites, using crimson hexagon's forsight social media analytics platform (hexagon, 2018), retrieving items that had at least one keyword in their headlines from the following list of words -{"gun", "firearm", "nra", "2nd amendment", "second amendment", "ar15", "assault weapon", "rifle", "brady act", "brady bill", "mass shooting"} -that have been translated into german, turkish, and arabic respectively by native speakers of the languages. in curating the multilingual datasets, we used the same set of frames as in gvfc. we then trained two native speaker coders for each language to apply the gvfc codebook proto-col for identifying frames and then measured their intercoder reliability (icr) in annotating a sample of 350, 200, and 210 german, turkish, and arabic news headlines, respectively. the coders achieve 92.6%, 98.5%, 78.1% agreement rates in identifying the first frame and 78.9%, 97.9%, 74.3% agreement rates for the second frame for german, turkish, and arabic samples. additionally, krippendorff's alpha for the 1st frame and the 2nd frame are 0.89, 0.66; 0.90, 0.74, and 0.69, 0.26 for german, turkish, and arabic, respectively. once a minimum of 70% agreement was reached, one coder of each language continued to code more headlines. annotation resulted in a total of 326, 100, and 388 non-duplicate headlines for german, turkish, and arabic. the average number of labels, i.e., label cardinalities, per headline are 1.4, 1.5, and 1.5, for german, turkish, and arabic, whereas it's 1.3 in gvfc, which is in english. as we can observe from the agreement rates, the arabic data has a relatively weaker icr, while the turkish data has the best icr. as high icr values imply that two coders consistently categorized the content similarly, they signal a high validity of the coded results. in turn, this is reflected in the performance of our model as it performs the worst in arabic (section 5). nonetheless, the quality of our curated data is substantially higher -the average of krippendorff's alpha is 0.82 -than contemporaries such as mfc (which is only in english) with an average alpha of less than 0.6 @xcite . ## model in this work, we extend the current state-of-theart model on the gvfc @xcite , which predicts only the first frame, into a multi-label approach and evaluate it across multiple languages. as previous work has showcased that bert surpasses lstm and gru-based architectures, we shift our focus in this work from architecture optimization to scalability of news framing analysis across multiple languages in a multi-label setting. bert relies on multiple stacks of the transformer's encoder blocks @xcite to learn vector representations of sentences. a single encoder block is composed of a self-attention layer followed by a fullyconnected layer. when a sentence -a sequence of tokens -is fed into the encoder, it passes through an embedding layer, a self-attention layer, and fullyconnected layers before being passed to the upper encoder block. the self-attention layer embodies three matrices called w q for the query, w k for the key, and w v for the value. each of these matrices is of size vocab_size × hidden_size, and thus each token in the vocabulary has its corresponding q, k, and v vectors. representations for each token are contextualized; namely, the representation of a token is the weighted average of all representations in the sequence. therefore, the vector representation for token x i is given by bert adds a special token for classification [cls] at the beginning of each sequence. then it learns the representation of this token and other tokens in the sequence by training on wikipedia corpus for two language tasks: next sentence prediction and masked language model (mlm), which was initially inspired by the cloze task @xcite . the contextual representation of the [cls] token encodes the syntactic and semantic constructs of the sequence, and one can fine-tune bert for various down-stream tasks. fine-tuning bert performs well on new tasks even with small datasets, which can be attributed to the data-efficient deep attention mechanism @xcite . the knowledge encoded within the vector representations of the tokens through pre-training also helps the classifier with the language understanding part of the task, reducing the need for a larger dataset. finally, a multilingual version of pre-trained bert, multibert, which is trained on the entire wikipedia dumps of 104 languages with the largest wikipedia, has recently been released, making it an excellent candidate for scaling to multiple languages. the multilingual pre-training and the utilization of sub-word tokenization allows multib-ert to represent sequences from any of these 104 languages @xcite and enables zero-shot classification on any of the languages (i.e., train on one language and test on another). in our case, since reproducing the effort put in gvfc, which was created by highly qualified journalism students in other languages, is prohibitive, employing a cross-lingual model such as multib-ert renders scaling to other words possible. ## experiments and results as input to our models, we follow previous work and rely on news headlines rather than news story content, due to reasons described by @xcite . to showcase the gains made on top of a multi-class approach by reformulating the problem as multi-label, we reproduce the method described by @xcite with both english bert and multiberts ( in for evaluation, we follow recent work and report macro and micro-averaged f1-scores @xcite , as well as exact-match (em) for samples which have single frames (em-1), two frames (em-2) and any number of frames (em-a). in note that, to accommodate multiple languages, we favor a multilingual language model. results in while being a state-of-the-art machine translation tool, google translate is the practitioner's handy translation guide, @xcite . in firstly, we translate gvfc from english, to target language @xmath0, ar, tr}, train multibert with ml focal loss and test on the tl. secondly, we use the english training set as is and translate target test sets to english. this latter setup lets us use engbert as well. we experiment with both cased and uncased models and observe that uncased performs better in de and ar. overall, we note that translating test sets to english results in better performance, which is intuitive as the model requires clarity in the language during training. all models in tables 3 and 2 use the same loss, and multibert experiments always use the cased version, following the authors' recommendation. we use 40 target samples of target language, translated to english, and include them in the training set to study few-shot performance. we only train the best performers, primarily based on f1 scores, among (3.2), (3.3) and (3.4), namely the models (3.3), (3.3) and (3.2) for de, ar and tr, respectively in ( furthermore, we compare zero-shot and fewshot performances of multibert when trained on original english versus code-switched train sets in notably, considering tables 2 and 3 together, a simple word-to-word translation for as little as 358 words, improves frame detection performance drastically even to the level of a complete translation of the test set to english. for turkish, code-switched training beats full translation of the test set into english in a few-shot setting; it results in a comparable performance for german and slightly worse predictions for arabic. we attribute the overall low performance for arabic to the relatively small icr in the annotation process. ## analysis to visualize our multi-label model we use the visualization tool by @xcite in figure 1 . in bert, every sequence is padded by a special classification token [cls] from the beginning. embedding generated for this token is used for classification into 9 classes. figures 1a and 1b demonstrate the attentions of this token to other tokens in the sequence. note that the given sample headline has indeed two frames i.e. "economic consequences" as the first and "public opinion" as the second. however, in a multiclass setup in which the model is configured to produce a single label, it learns to disregard the second frame "public opinion" while strongly attending the words "fargo" and "credit" related to for the theme of "economic consequences". on the contrary, a multi-label model correctly attends all words that are related to both frames i.e. "fargo", "credit", "nuns" and "opposition" and predicts "economic consequences" and "public opinion" correctly. another interesting observation is related to bias induced by translation. in german, the phrase "schärferes waffenrecht" means "stricter gun regulation". however, google translate translates half of the headlines that include the expression as "stricter/sharper gun rights" which makes the model predict "gun rights" rather than "gun control" as the frame. a discrepancy like this is widely deceptive and jeopardizes the learning, whether it happens in the training or validation set. however, in code-switched training, one has better control over the translation, as one only translates a manageable number of words. we observe that code-switched training escapes this bias through correctly translated keywords "gun" and "laws" to german. additionally, we find our models catching several annotation errors such as the headline in turkish "obama'dan lgbti bireylerin gittigi bir kulüpte 49 kişiyi öldüren orlando saldırganı hakkında açıklama" which translates as "obama gave a statement about the orlando shooter who killed 49 in an lgbti club." is annotated as "politics"." in contrast, the model predicts "society/culture" and "politics", attending to "lgbti" and "club". ## conclusion in this work, we present a novel code-switch model for the task of automatic cross-lingual news frame detection and show that it matches the performance of full translation if not overrides. moreover, we leverage an existing dataset by making use of multiple labels, create benchmark news framing test sets for three new languages, and employ a variant of focal loss to account for class imbalance in the data. in conclusion, while accounting for multiple frames per sample, we demonstrate how a crosslingual analysis of news framing is informative and insightful in developing a global view surrounding the gun violence problem in the u.s.