CN - HIT - MI . T at S em E val-2020 Task 8: Memotion Analysis Based on BERT (2020)

Abstract: Internet memes emotion recognition is focused by many researchers. In this paper, we adopt BERT and ResNet for evaluation of detecting the emotions of Internet memes. We focus on solving the problem of data imbalance and data contains noise. We use RandAugment to enhance the data of the picture, and use Training Signal Annealing (TSA) to solve the impact of the imbalance of the label. At the same time, a new loss function is designed to ensure that the model is not affected by input noise which will improve the robustness of the model. We participated in sub-task a and our model based on BERT obtains 34.58% macro F1 score, ranking 10/32.

## introduction memes are from people's culture or some social activities in daily life, usually composed of one or two forms of image, video, gif, and text @xcite . memes are active in people's social media, but with the number of memes increasing, offensive memes are also increasing @xcite . so for many social media companies, how to identify the type of meme is very important. using machine learning to identify the type of meme is a very important solution, and has shown promising performance. in semeval-2020 task 8: memotion analysis @xcite , the organizer collecting the internet memes and then using ocr recognition it and correct manually the text on the pictures. the task is divided into three sub-tasks: a) sub-task a detects the emotions of the internet memes, which are divided into positive, negative and neutral; b) sub-task b detects the types of humor expressed by internet memes, which are divided into sarcastic, humorous, offensive and motivation type, a meme can belong to multiple categories; c) sub-task c is a semantic level classification of the various humor types of task b, each type is divided into 4 categories: not, slightly, mildly and very. the task uses macro f1 as the evaluation standard for sub-task a, and average macro f1 as the evaluation standard for sub-tasks b and c. there are three challenges for this task. the first, how to fuse picture and text features, because whether it is a separate text or picture, the meaning expressed is lacking, and it is necessary to fuse the picture features and text semantics to obtain the best classification results. the second, the distribution of train data is imbalanced, which will increase the learn difficulty of the model. at last, the text information provided by the training data contains too much noise, which affects the semantic understanding of the text by the model and it will need to clean the data. because we didn't find a proper way to fuse text and picture feature, we used bert @xcite for text feature extraction, and resnet @xcite for image feature extraction. through experiments, we found that the macro f1 score of bert was higher than resnet, so we submitted the predicted result of bert in the final submission result. in the rest of this article, we organize the content as follows: section 2 introduces the background of this task; section 3 mainly introduces the model we used and some details; section 4 introduces the data and preprocessing; section 5 shows the experimental results and make some analysis. finally, we discussed our work and future work directions. we participated in sub-task a, which is to judge the emotions of memes, and we only used the data provided by the competition organizer @xcite . in order to effectively extract the image features and text features, we considered using the single-modal model to extract the features of the image and the text, and then the multimodal model is designed on the basis of the single-modal model. however, the multimodal model is found to be less effective than the single-modal model. it will discuss in section 5. ## system overview in this task, we mainly used the resnet-101 pre-training model and the bert pre-training model. we refer to vistanet @xcite , hierarchical fusion model @xcite and tensor fusion network @xcite respectively. we also refer to @xcite , and extract the effective features of the picture through the sentences in the text, and then merge the text features and the effective features of the picture. but we found that the prediction results of the multimodal model are not as good as the single-modal model, so we did not use multimodal model in the final submission results, see section 5 for details. ## results the evaluation metric of sub-task a is macro-f1. we divide the dataset into training (5595 meme data) and development (1397 meme data) subsets. we initialize the model with different random seeds for many times. various models are used for training. at the same time, we use a variety of models for training, and the prediction results are shown in from the results, we can see that the prediction results of the multimodal model hfm and tfnet are not as good as the single-modal model. we tried vistanet @xcite , hfm @xcite and tfnet @xcite . vistanet's prediction result in this task is the worst. we think vistanet's main purpose is to filter important text through multiple pictures. it focuses is not just the fusion of text and picture information, so we gave it up early. however, the prediction results of hfm and tfnet are not as good as the single-modal model too. by observing the data set , we found that the meme dataset @xcite of semeval-2020 task8 has a feature, that is the understanding of meme is very dependent on the alignment of text segment and image regions. but neither hfm nor tfnet focuses on it. so we later designed a model to align the picture and the text. we divide the text into sentences and get the features of sentences by bert @xcite . then we get the 7x7 image regions through resnet-101. at last, performing attention operations on the images regions by sentences. but the actual effect is not good, which is worse than hfm and tfnet. we found that in many cases, a text is only divided into one sentence, so we consider that it may be the sentence level division is too rough. we guess that the use of phrase level or word level division may achieve better results, but we have not designed a suitable model. so in the end we did not use the multimodal model. the confusion matrix is shown in figure 1 . from the figure, we can see that because of the problem of label imbalance in the dataset, the precision of the negative label is only 20%, and the precision of the positive label reaches 61.13%. this also leads to this macro-f1 is relatively low.