Prediction of People’s Emotional Response towards Multi-modal News (2022)

Abstract: We aim to develop methods for understanding how multimedia news exposure can affect people’s emotional responses, and we especially focus on news content related to gun violence, a very important yet polarizing issue in the U.S. We created the dataset NEmo+ by significantly extending the U.S. gun violence news-to-emotions dataset, BU-NEmo, from 320 to 1,297 news headline and lead image pairings and collecting 38,910 annotations in a large crowdsourcing experiment. In curating the NEmo+ dataset, we developed methods to identify news items that will trigger similar versus divergent emotional responses. For news items that trigger similar emotional responses, we compiled them into the NEmo+-Consensus dataset. We benchmark models on this dataset that predict a person’s dominant emotional response toward the target news item (single-label prediction). On the full NEmo+ dataset, containing news items that would lead to both differing and similar emotional responses, we also benchmark models for the novel task of predicting the distribution of evoked emotional responses in humans when presented with multi-modal news content. Our single-label and multi-label prediction models outperform baselines by large margins across several metrics.

## introduction understanding how exposure to certain textual and visual news affects people's emotional reactions is important for detecting, educating, and correcting intentional or unintentional emotional manipulation of readers. as a step towards detecting such manipulations and raising news consumers' visual literacy, in this work we develop methods for predicting emotional responses towards news headlines and images. to the best of our knowledge, machine learning tools that predict how a reader will react emotionally to a certain news headline, choice of a lead image, or combination of both do not exist. in this paper, we introduce tools that enable such prediction and thus can shed light on effects of news presentation, which is important to both editors and consumers of news. the dataset we utilize in this work has been developed in phases. it first started with the headlines of news articles in the gun violence frame corpus (gvfc) @xcite , along with corresponding lead images of these articles @xcite . a previous study started a crowd-sourcing experiment to collect emotional response annotations to the news headlines and images, producing the bu-nemo dataset @xcite . in this work, we extend the above emotional response experiment significantly. we utilize our new expanded dataset, named nemo + , and present the first benchmark of models to predict the evoked emotional responses in news consumers when presented with multi-modal news content. ## method we benchmark machine learning models, described in detail in section 5, for each of the three conditions (t, i, ti) to examine whether text or image when presented separately or together provide more context and help viewers form an emotional response towards particular news content. ## results we split the datasets into train / validation / test sets in the ratio of 50%:25%:25% and all of experiment results are reported on the test set. ## limitations & future work there exist some limitations to our work. firstly, the multi-modal classification model we benchmark in the ti condition has exhibited lower performance than in the t and i condition ( one limitation with our multi-label experiment is that the conversion to binary labels causes a loss in relative scale of information among the n labels emotional categories. an alternative approach to this problem in future work could be to model the distribution of emotions for each news sample with a kl-divergence loss instead. in future work, we could also derive deeper insights by using the intensity scores we collected in section 3.1 to predict the strength of emotional responses to news. another future task is to predict whether a given news headline and/or image will elicit emotional consensus, or result in a divided response among readers. it will also be interesting to study the relationship between emotional responses and the framing of the news and to extend the task to multilingual setting @xcite . finally, we are interested in making the benchmarked systems for predicting emotional responses to news accessible to researchers from a diverse array of disciplines (in similar fashion to the interactive computational framing website: open-framing @xcite ) so that researchers from various disciplines can conduct further studies on the potential benefits and risks of such system. ## conclusion we have shown that we can effectively, to some degree, predict the emotional response to news headline and image using standard text-and vision-classification models. our work is the novel attempt at benchmarking the task of predicting how exposure to certain textual and visual news affects people's emotional reactions. this task has wide implications for both news consumers and news professionals. potential misuses are the possibilities that our tool can be intentionally used to predict and manipulate the emotional reactions of news consumers with specific choices of news headlines and images. however, news editors could aim to avoid sensationalizing their produced media content by using prediction systems like ours. this would be useful in situations where presentation of sensitive news topics (war crimes, terror, etc.) benefits from a more informed selection of image-to-text combinations that can convey important information over sensational, distracting content. publishers and experts can use this tool to recognize and avoid emotionally-manipulative content. social media platforms could also use insights on evoked emotion from media in order to predict whether a post is likely to be click-bait. educators could also use our system for teaching visual media literacy.