Detecting Frames in News Headlines and Lead Images in U . S . Gun Violence Coverage (2021)

Abstract: News media structure their reporting of events or issues using certain perspectives. When describing an incident involving gun violence, for example, some journalists may focus on mental health or gun regulation, while others may emphasize the discussion of gun rights. Such perspectives are called “frames” in communication research. We study, for the first time, the value of combining lead images and their contextual information with text to identify the frame of a given news article. We observe that using multiple modes of information(article- and image-derived features) improves prediction of news frames over any single mode of information when the images are relevant to the frames of the headlines. We also observe that frame image relevance is related to the ease of conveying frames via images, which we call frame concreteness. Additionally, we release the first multimodal news framing dataset related to gun violence in the U.S., curated and annotated by communication researchers. The dataset will allow researchers to further examine the use of multiple information modalities for studying media framing.

## introduction media framing refers to the journalistic practice of selecting aspects of a perceived reality and making them more salient in news coverage @xcite . in political communication, for example, news framing is helpful as it reveals how the news article is structured to promote a certain side of the political spectrum, thus influencing the public opinion in a particular way. journalists have been using both text and images to frame news stories. images in news stories can help convey controversial or provocative meanings that would otherwise be unpalatable to the news audience, if it were spelled out in text @xcite . while text is more influential in changing opinions, visuals elicit more attention and emotional reactions, resulting in behavioral change @xcite @xcite . lead images may carry additional background knowledge about the event (e.g., showing well-known people and locations). an image showing a school, for example, might suggest an article about gun violence focuses on the "school/public safety" frame. text and images thus work in tandem to create a holistic perception of news and must be considered together when analyzing news frames @xcite . given the importance of visuals in media framing and the rising gun violence in the u.s. @xcite , we extend the gun violence frame corpus (gvfc) @xcite , which contains news headlines related to u.s. gun violence and their domain-expert frame annotations, by retrieving the lead images of the articles and obtaining their relevance annotations from communication domain experts (i.e., an image is annotated as relevant if it expresses the annotated headline frames). notably, about half of the time, the images presented do not express the annotated headline frames ( in this work, we comprehensively explore the use of multimodal information from news articles i.e., headlines or summaries, and their lead images i.e., the images, categories of objects in the images, or the background/real-world knowledge contained in them, for predicting frames. our results show that for news articles with relevant images, using only image-derived features or only articlederived features (i.e., headlines and/or extractive summaries) yields less accurate frame predictions than our multiple modalities approach. when considering articles with irrelevant images, the accuracy of the multimodal approach is comparable to that based on only article-derived features. we also observe that adding image contextual information using the google web entity tagger api 1 or an entity-aware news image caption generation model @xcite or by asking humans to annotate the central subject of the image in terms of pre-defined categories that cue gun violence frames, such as politician (politics frame), legislative buildings (gun control frame), school/campus (school/public safety frame), etc., improves the performance of frame prediction, compared to using raw images alone. the api tags capture background information associated with an image from the web, such as the list of named entities in the image, by finding similar images in the web and parsing the associated web page contents. news image captions capture real-world information in the image, e.g., the names of people and objects, by learning to associate words in the article text with faces and objects. human annotations of the central subject of the image in terms of categories such school or legislative buildings, capture the annotators' background knowledge of the identities of entities in the image. overall, our contributions are the following: (1) a well-curated multimodal text-image framing dataset with expert annotations 2 : with the goal of predicting frames based on multiple information modalities, we augment gvfc by using the article urls to retrieve lead images of articles and annotate the images for their visual framing labels, which include (a) the subject/race/ethnicity (sre) annotations of the central subject of the image (i.e., suspect vs. victim vs. politician, etc.) and whether the image contains anything related 1 https://cloud.google.com/vision/docs/ detecting-web 2 we include the dataset in the supplementary material and the multimodal annotation is available for download through the gvfc dataset website: https://derrywijaya. github.io/gvfc.html . to race/ethnicity, and (b) the image relevance annotations i.e., whether the images are relevant to the annotated frames of their headlines. in addition to frame annotations of news article headlines and lead images, for each image we provide its url, web entity tag (api tag), caption generated using a state-of-the-art news image captioning system @xcite , and the article summary generated by a state-of-the-art extractive summarization system @xcite . (2) comprehensive study and development of methods to combine multimodal information to predict article frames and image relevance: we explore various approaches to predict image relevance and article frames using information from both article and lead images, using bert @xcite to represent text and a deep convolutional neural network resnet-50 @xcite to represent raw images. (3) frame concreteness: we propose a novel method for measuring the ease of conveying frames through images via the concreteness of words in its headlines, i.e., the ease of identifying tangible concepts and mental images that arise in correspondence to words @xcite , and relate frame image relevance to frame concreteness. ## related work media framing is related to many factors such as word choice, the presentation of background information, and the emphasis on certain actors. the subtle nature of news framing can influence the opinion of readers in a certain way without them even noticing it, hence its analysis has many implications, e.g., it has been used to understand why important public affairs issues such as gun violence are polarizing @xcite , how media manipulation strategies are conducted @xcite , or how framing is used to perpetuate racial biases @xcite . in journalism, most "framing analyses" have been done manually hence they are not scalable @xcite . in natural language processing (nlp), automated frame detection focuses on predicting frames from news texts. some of these methods rely on topic models @xcite . @xcite devised various deep neural networks (lstms, bilstms, or grus) for predicting frames at the sentence level in the media frame corpus (mfc) @xcite . the most recent method for predicting news frames in headlines is the work of @xcite ; @xcite . they fine-tuned bert @xcite using focal loss @xcite to predict frames of headlines. they also released a framing benchmark dataset, the gun violence frame corpus (gvfc), of news links and headlines with frame annotations, related to gun violence in the u.s. they show that fine-tuning bert for frame prediction using news headlines results to significantly higher accuracy than previous methods. in this paper, we use gvfc articles' lead images and perform experiments with a rich set of unimodal and multimodal information to predict frames. although images and text have been used together as multimodal inputs to improve performance in other nlp tasks such as machine translation @xcite @xcite @xcite or in vision-language tasks such as multilingual image retrieval or captioning @xcite @xcite , the use of images and text in tandem to automatically analyze framing i.e., computational multimodal framing has never been explored-all previous works in multimodal framing have been conducted manually @xcite @xcite @xcite . given the growing importance of visual journalism and the contribution of images to media framing which suggest that images may be able to help interpret text, our work is the first to conduct computational multimodal framing analysis, which will enable scalable multimodal framing analysis. ## dataset our multimodal version of gvfc contains news headlines and their corresponding lead images, news urls, and the entire news text. the lead images are either the pictures shown at the top of news articles or the editor-picked thumbnails that are shown in news services such as google news (fig. 1 ). using brandwatch consumer research foot_0 we analyze 3,000 news headlines, of which 1,300 are are annotated with 9 major frames, e.g., politics, gun control/regulation, mental health, race/ethnicity, etc. ( the details of the visual annotation codebook used to train the coders are given in the appendix and supplementary material. the coders' agreement on how to apply the codes is measured with inter-coder reliability (icr). high icr values (above 90% agreement or 0.70 krippendorff α @xcite ) imply that two or more coders consistently categorized the content similarly, signaling a high validity of the results. in our dataset, icr was met on all variables: subject (90% agreement, 0.88 α), race/ethnicity (91% agreement, 0.64 α) and relevance (88% agreement, 0.75 α). the number and ratio of relevant images per frame are shown in ## experiments we experiment with unimodal ( §4.1) and multimodal ( §4.2) information obtained from each article and its lead image. we train and report 4-fold cross validation frame prediction accuracies ( ## discussion of results despite the challenges of a highly nuanced multiclass frame identification and an intrinsically imbalanced dataset, we achieve a high prediction accuracy of up to 87% for articles with relevant images, and 82.4% for all articles ( it is instructive to examine the utility of articleand image-derived features, and a fusion of all in cases where the lead image is relevant to the article headline. we observe that contextual information derived from the image, in the form of api tags or a caption, along with the article headline, can drive the article perspective more clearly. the headline + api tags combination provides the best performance (87%), compared to image-only (43%), the sre image annotation (81.2%), or the the article headline (83%) which is a strong unimodal baseline. even when considering examples with irrelevant images, adding api to headlines does not hurt performance and is comparable to using headlines alone, which is unlike sre whose performance drops significantly for articles with irrelevant images as these annotations are designed with relevant, i.e., frame-implying images in mind. furthermore, sre requires training another model to produce these annotations automatically, which is not trivial as they capture real-world knowledge of the subjects in the image, e.g., whether the person in the image is a politician (cuing politics frame) or a gun activist/nra representative (cuing 2nd amendment). these findings, namely that api tags, captions, or sre yield higher accuracy than the raw image alone, indicate the importance of the contextual or background knowledge of the lead image in driving the news frame this strongly suggests that the highly nuanced task of frame prediction is challenging using images in isolation. our findings also confirm previous observations that training with multiple input modalities, e.g., both visual and textual inputs is hard as each modality may generalize differently and hence underperform when trained jointly @xcite . in terms of relevance prediction, the performance is highest for models supplied with frame labels, mimicking the relevance annotation process. given a headline and a frame, our method can correctly predict the relevance of an image to the frame with 74% accuracy using the image's api tag. without frame labels, however, the accuracy of the top-performing method drops to 68.1% and is based on sre only. while several sre categories are strong indicators for certain frames, e.g., the presence of demonstrators suggests the public opinion frame, the use of sre at inference time necessitates the training of another model for predicting these annotations, which is not trivial. to measure concreteness of frames and test this hypothesis, we trained a regression network that takes as input a word's vector representation as extracted and concatenated from the last 4 layers of the pre-trained bert model and outputs its concreteness measure between 1 (most abstract) to 5 (most concrete). we use a dataset created by @xcite , which contains human evaluations of concreteness for 39,954 english words, to train and evaluate the network, achieving a high 0.95 pearson's correlation between our concreteness predictions and the ground-truth measures. the concreteness of a frame is then measured as the average concreteness of non named-entity words in its headlines (we treat named-entities as having a concreteness measure of 5). as seen in figure 2 , no frame has a high (> 4) average concreteness. we observe, however, that some of the more concrete frames have higher ratios of their images error type description examples plausible interpetation predicted frames can be appropriate labels. example frame: mental health headline: florida shooter a troubled loner with white supremacist ties • model erroneously predicted "race/ethnicity": "white" + api tag: nikolas cruz stoneman douglas high school shooting marjory stoneman douglas murder mass shooting ar-15 style rifle suspect student • model correctly predicted "mental health": "suspect student" inferring frames predicted frames capture an author's intention example frame: 2nd amendment/gun rights not explicitly cued in text without sufficient text evidence. headline: the nra versus the constitution • model erroneously predicted "gun control": "nra" + api tag: pennsylvania obergefell v. hodges supreme court of the united states concealed carry rights reciprocity act of 2017 constitution of the united states • model correctly predicted "2nd amendment": "rights","constitution" missing necessary frames can be directly cued by lexical items example frame: politics contextual knowledge (e.g. politicians' names cue politics frame), headline: in closed-door meeting, roskam brings pro-gun rights teens to talk gun violence yet the model lacks real-world knowledge prevention with students required to identify those. peter roskam republican party democratic party united states congress illinois member of congress 2018 united states elections united states house of representatives presidency of donald trump house committee on ways and means • model correctly predicted "politics": using lexical cues of politician and party names and political terms: "trump", "republican", "democratic", "congress" overgeneralizing highly correlated words and phrases example frame: race/ethnicity highly-correlated features that do not directly cue frames, headline: lawyers call us gun charges for mexican man "vindictive" along with and are used in different contexts. • model erroneously predicted "mental health": "vindictive" (long-distance dependencies) + api tag: shooting of kate steinle acquittal murder san francisco homicide jury death defendant illegal immigration manslaughter • model correctly predicted "race/ethnicity": better context we also report per frame classification f1 scores for the baseline and our best performing models on articles with relevant images and all articles, i.e., when using the article headline alone, or with the the api, the caption, and the summary in figure 3 . performance when using information from articles and images is remarkable for frames with either high image relevance ratios or high concreteness. in articles with relevant images, the frames with the highest image relevance ratio i.e., "politics" shows an impressive 96.6% f1 score with the headline and the api tags, followed by "public opinion" (87.1%); while frames with few relevant images ("2nd amendment", "society/culture"), have a substantially lower f1 scores. on all articles, the inclusion of articles with irrelevant images can hurt performance for frames with high image relevance ratio such as politics and public opinion. however, other frames may benefit from having more examples to learn from. for example, we observe that a low image relevance but a highly concrete frame such as "race/ethnicity" benefits significantly from this inclusion, reaching a high f1 score of 90.6% using all articles from 82.1% using only articles with relevant images as the model learns more lexical cues, e.g., named-entities from headlines of more articles, including those with irrelevant images. concreteness may also augment relevance in explaining improved performance for some frames on all articles. we observe high correlations between frame average concreteness and average f1 scores on articles with relevant images (pearson correlation of 0.93) and on all articles (pearson correlation of 0.94), which exceed correlations between frame image relevance ratio and average f1 scores (pearson correlation of 0.81 and 0.67 on articles with relevant images and on all articles, respectively). these findings suggest that concreteness might be worth exploring for frame prediction and use of imagery in the future, in addition to concreteness annotation in framing datasets. to complete our analysis, we applied the frame prediction error taxonomy proposed by @xcite to our news framing with imageand article-derived information, to identify and summarize common classification errors in ## conclusions we presented the first ever study and dataset on computational multimodal framing. our results show that image-derived contextual features can be useful for providing missing contextual or background information that can improve frame prediction significantly, particularly for concrete frames or frames with relevant images. we also proposed methods for predicting frame image relevance and for measuring frame concreteness, which we define as the ease of expressing frames via images.