On the Complementarity of Images and Text for the Expression of Emotions in Social Media (2022)

Abstract: Authors of posts in social media communicate their emotions and what causes them with text and images. While there is work on emotion and stimulus detection for each modality separately, it is yet unknown if the modalities contain complementary emotion information in social media. We aim at filling this research gap and contribute a novel, annotated corpus of English multimodal Reddit posts. On this resource, we develop models to automatically detect the relation between image and text, an emotion stimulus category and the emotion class. We evaluate if these tasks require both modalities and find for the image–text relations, that text alone is sufficient for most categories (complementary, illustrative, opposing): the information in the text allows to predict if an image is required for emotion understanding. The emotions of anger and sadness are best predicted with a multimodal model, while text alone is sufficient for disgust, joy, and surprise. Stimuli depicted by objects, animals, food, or a person are best predicted by image-only models, while multimodal mod- els are most effective on art, events, memes, places, or screenshots.

## introduction the main task in emotion analysis in natural language processing is emotion classification into predefined sets of emotion categories, for instance, corresponding to basic emotions (fear, anger, joy, sadness, surprise, disgust, anticipation, and trust, @xcite . in psychology, emotions are commonly considered a reaction to an event which consists of a synchronized change of organismic subsystems, namely neurophysiological changes, reactions, action tendencies, the subjective feeling, and a cognitive appraisal @xcite . these theories recently received increasing attention, for instance, by comparing the way how emotions are expressed, based on these components @xcite , and by modelling emotions in dimensional models of affect @xcite or appraisal @xcite . further, the acknowledgment of emotions as a reaction to some relevant event @xcite leads to the development of stimulus detection systems. this task is formulated in a tokenlabeling setup @xcite @xcite @xcite , as clause classification @xcite @xcite @xcite , or as a classification task into a predefined inventory of relevant stimuli @xcite . in social media, users express emotions including text and images. most attention has been devoted to twitter, due to its easy-to-use api and popularity @xcite @xcite . however, this platform has a tendency to be text-focused, and has therefore not triggered too much attention towards other modalities. although text may be informative enough to recognize an emotion in many cases, images may modulate the meaning, or sometimes solely convey the emotion itself (see examples in figure 1 ). the growing popularity of vision-centered platforms like tiktok or instagram, and lack of research on multimodal social media constitute a research gap. with this paper, we study how users on social media make use of images and text jointly to communicate their emotion and the stimulus of that emotion. we assume that linking depictions of stimuli to the text supports emotion recognition across modalities. we study multimodal posts on the social media platform reddit foot_0 , given its wide adoption, the frequently found use of images and text, and the available programming interfaces to access the data @xcite . our goal is to understand how users choose to use an image in addition to text, and the role of the relation, the emotion, and the stimulus for this decision. further we analyze if the classification performance benefits from a joint model across modalities. figure 1 shows examples for reddit posts. in figure 1a , both image and text would presumably allow to infer the correct emotion even when considered in isolation. in figure 1b , additional knowledge of the complementary role of the picture depicting an animal can inform an emotion recognition model. in figure 1c the image alone would not be sufficient to infer the emotion, but the text alone is. we therefore contribute (1) a new corpus of multimodal emotional posts from reddit, which is annotated for authors' emotions, image-text relations, and emotion stimuli. we (2) analyze the relations of the annotated classes and find that certain emotions are likely to appear with certain relations and emotion stimuli. further, we (3) use a transformer-based language model (pretrained roberta model, @xcite and a residual neural network (resnet50, @xcite to create classification models for the prediction of each of the three classes mentioned above. we analyze for which classification tasks multimodal models show an improvement over unimodal models. our corpus is publicly available at https://www.ims.uni-stuttgart.de/data/mmemo . one of the questions that needs to be answered when developing an emotion classification system is that of the appropriate set of emotions. there are two main theories regarding emotion models in psychology that found application in nlp: discrete sets of emotions and dimensional models. psychological models that provide discrete sets of emotions include ekman's model of basic emotions (anger, disgust, surprise, joy, sadness, and fear, @xcite ) and plutchik's wheel of emotions (adding trust and anticipation, @xcite . dimensional models define where emotions lie in a vector space in which the dimensions have another meaning, including affect @xcite and cognitive event appraisal @xcite @xcite . in our study, we use the eight emotions from the plutchik's wheel of emotions. multimodal analyses. the area of emotion analysis also received attention from the computer vi-sion community. a common approach is to use transfer learning from general image classifiers @xcite or the analysis of facial emotion expressions, with features of muscle movement @xcite or deep learning @xcite . @xcite use texture and color features to analyze social media content. other useful properties of images for emotion analysis include the occurrence of people, faces, shapes of objects, and color distributions @xcite . such in-depth analyses are related to stimulus detection. @xcite detect emotion-eliciting image regions. they show, on a flickr image dataset, that not only objects @xcite and salient regions @xcite have an impact on elicited emotions, but also contextual background. @xcite , inter alia, show that it is beneficial for emotion classification to explicitly integrate visual information from emotion-eliciting regions. similarly, @xcite study the relationship between emotion-eliciting image content and human visual attention. image-text relation. a set of work aimed at understanding the relation between images and text. @xcite establish a taxonomy of 49 functions of illustrations relative to text in us government publications. the relations contain categories like "elicit emotion", "motivate", "explains", or "compares" and "contrasts". @xcite aim at understanding both the role of an image and of text. in contrast to these studies which did not develop machine learning approaches, @xcite develop automatic classification methods for detection of relations between the image and a slogan in advertisments. they detect if the image and the text make the same point, if one modality is unclear without the other, if the modalities, when considered separately, imply opposing ideas, and if one of the modalities is sufficient to convey the message. @xcite focus on detecting if captions of images contain complementary information. @xcite infer relationship categories between the text and image of twitter posts to see how the meaning of the entire tweet is composed. @xcite focus on understanding the intent of the author of an instagram post and develop a hierarchy of classes, namely advocative, promotive, exhibitionist, expressive, informative, entertainment, provoca-tive/discrimination, and provocative/controversial. they also analyze the relation between the modalities with the classes divergent, additive, or parallel. our work is similar to the two previously mentioned papers, as the detection which emotion is expressed with a post is related to intent understanding. ## corpus creation to study the roles of images in social media posts, we create an annotated reddit dataset with labels of emotions, text-image relations, and emotion stimuli. we first discuss our label sets and then explain the data collection and annotation procedures. ## methods in the following, we present the models that we used to predict (1) each variable (emotion, stimulus, relation) separately in each modality, and (2) across modalities with joint models. ## results we evaluate our models on predicting emotions, text-image relations, and emotion stimuli using unimodal and multimodal models, based on the f 1 measure. we use the dataset of 1054 instances in which we aggregate the labels from the three annotators by accepting a label if one annotator assigned model emo. rel. stim. majority baseline .22 .56 .21 unimodal text .53 .77 .45 image .41 .67 .59 multimodal early fusion .40 .72 .33 late fusion .47 .72 .41 model-based fusion .53 .76 .63 in other words, the annotation process requires annotators to select a single label (for each set of labels), e.g. one emotion per post; however, the experiments are conducted using multiple labels per set, depending on how many labels are given by three annotators for each set of labels. the data is randomly split into 853 instances for training, 95 instances for validation, and 106 test instances. for most stimulus categories, either the image or a multimodal model performs best. this is not surprising, given that the stimulus is often depicted in the visual part of a multimodal post. more complex depictions that could receive various evaluations, like art, events/situations, and memes require multimodal information. in those, the image information alone is not sufficient-the performance difference is between 22pp and 13pp in f 1 . for those stimuli, in which the text-based model outperforms the multimodal models, the difference is lower. the text-based model is never performing best, but shows acceptable performance for animals, memes, screenshots and person depictions. regarding the relations, the complementary class is predicted with the best performance; which is due to the frequency of this class. the label decorative can only be predicted with a (slightly) acceptable performance with the multimodal approach, while illustrative predictions based on text-only are nearly en par with a multimodal model. from the three multimodal fusion approaches, early fusion performs the worst, followed by late fusion. model-based fusion most often leads to the best result. we show examples for instances in which the multimodal model performs better than unimodal models in ## conclusion and future work with this paper, we presented the first study on how users in social media make use of text and images to communicate their emotions. we have seen that the number of multimodal posts in which the image does not contribute additional information over the text is in the minority, and, hence, interpretation of images in addition to the text is important. while the inter-annotator agreement for relation was not reliable enough to draw this conclusion, prediction of stimulus correlates with prediction of emotion due to the information that is present in the image but missing in the text, and thus makes images play a significant role in analysis of social media posts. this is also the first study on stimulus detection in multimodal posts, and we have seen that for the majority of stimulus categories, the information in the text is not sufficient. in contrast to most work on emotion stimulus and cause detection in nlp, we treated this task as a discrete classification task, similar to early work in targeted sentiment analysis. an interesting step in the future will be to join segment-based open domain stimulus detection, as it is common in text analysis, with region-based image analysis, and ground the textual references in the image. this will allow to go beyond predefined categories. qualification description qualification test 1: emotional/nonemotional posts 5 posts presented to annotators to label the post emotional or non-emotional; passing score of 80% qualification test 2: emotion, relation, stimulus identification 5 posts presented to annotators to label the post for emotions, relations, and stimuli; passing score of 80% region annotators must reside in either of the six english-speaking countries (australia, canada, ireland, new zealand, united kingdom, united states) to force the task to be done by native speakers. human intelligence task (hit) approval rate the hit approval rate represents the proportion of completed tasks that are approved by requesters and ensures the quality of the job workers do on the platform.