Hitachi at S em E val-2020 Task 8: Simple but Effective Modality Ensemble for Meme Emotion Recognition (2020)

Abstract: Users of social networking services often share their emotions via multi-modal content, usually images paired with text embedded in them. SemEval-2020 task 8, Memotion Analysis, aims at automatically recognizing these emotions of so-called internet memes. In this paper, we propose a simple but effective Modality Ensemble that incorporates visual and textual deep-learning models, which are independently trained, rather than providing a single multi-modal joint network. To this end, we first fine-tune four pre-trained visual models (i.e., Inception-ResNet, PolyNet, SENet, and PNASNet) and four textual models (i.e., BERT, GPT-2, Transformer-XL, and XLNet). Then, we fuse their predictions with ensemble methods to effectively capture cross-modal correlations. The experiments performed on dev-set show that both visual and textual features aided each other, especially in subtask-C, and consequently, our system ranked 2nd on subtask-C.

## introduction recently, internet memes -visual plus textual content on the internet -have been widely spreading due to the rapid growth of social networks, and thus, recognizing the emotions of memes is required to analyze social interactions. in semeval-2020 task 8: memotion analysis @xcite , we aim at automatically recognizing the various emotions of memes. the task contains three subtasks: subtask-a, where participants are required to predict the sentiment of a given meme, subtask-b, to predict whether a given meme represents emotions expressing certain aspects (i.e., humorous, sarcastic, offensive, and motivational) or not, and subtask-c, to predict a four graded degree (i.e., 0, 1, 2, or 3) to which a meme represents the emotions of the above aspects. the challenge to deal with in the above tasks is how we can incorporate both visual and textual impressions. to this end, we propose a simple ensemble of strong pre-trained models of single modality to capture cross-modal correlations, as shown in figure 1 . to the best of our knowledge, ensembles with strong pre-trained models from different modalities have hardly been explored because multi-modal systems such as visual questions and answers @xcite focus mostly on multimodal unified models. from this perspective, our method would provide a simple but effective approach to dealing with both visual and textual features at once. experimental results show that modality ensemble works well for subtask-b and subtask-c, showing the effectiveness of our proposed system. the experiments performed on dev-set also show that both visual and textual features aid each other, especially in subtask-c, and consequently our system ranked 2nd on subtask-c. ## background recent years have seen advances in the automatic recognition of visual plus textual content @xcite . @xcite defined a multi-modal task called vi-textual models visual models final prediction (e.g., "motivational") modality ensemble pnasnet-5 senet inception-resnet polynet bert xlnet gpt-2 transformer-xl let's party sual question answering (vqa), where paired images and questions (in natural language) are supplied. answering the questions requires an understanding of vision and language and a bit of commonsense knowledge. given that the questions of the vqa dataset have some bias, @xcite released a new dataset called gqa, which avoids bias by automatically generating a variety of questions from scene graphs. while these tasks aim rather at understanding the contents of images (ex., the objects, their colors, their spatial relations, or some implications they have), @xcite defined a new task, memotion analysis, aiming at automatically recognizing the emotions attached to the contents by the creator. we tackle this task by leveraging strong single-modal pre-trained models and fusing them to capture cross-modal correlations. ## task setup for all the subtasks, the inputs are the same, i.e., pairs of an image and a piece of text ("memes" in short). the details of each subtask are as follows. note that all the subtasks are classification problems on for given memes. in subtask-b and subtask-c, we solved single emotion-type classification problems separately, rather than building unified models for all the emotion types. given meme images, we train single-modal models (i.e., either textual or visual) for each single-emotion classification problem. then, we just aggregate all scores of the single-modal models as the input of the ensemble models and achieve the final outputs from the models. ## conclusion in this paper, we presented a simple but effective modality ensemble for predicting multi-modal internet meme emotions. for both visual and textual modalities, we fine-tuned strong pre-trained models independently. in addition, we fused the predictions with an ensemble method to capture cross-modal correlations. the experiments on the dev-set show the promising results of our strategy. we will explore a more effective way of handling the multi-modality of an internet meme.