YNU - HPCC at S em E val-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT (2022)

Abstract: This paper describes our system used in the SemEval-2022 Task5 Multimedia Automatic Misogyny Identification (MAMI). This task is to use the provided text-image pairs to classify emotions. In this paper, We propose a multi-label emotion classification model based on pre-trained LXMERT. We use Faster-RCNN to extract visual representation and utilize LXMERT’s cross-attention for multi-modal alignment. Then we use the Bilinear-interaction layer to fuse these features. Our experimental results surpass the F_1 score of baseline. For Sub-task A, our F_1 score is 0.662 and Sub-task B’s F_1 score is 0.633. The code of this study is available on GitHub.

## introduction in social networks, meme is mainly used to express the emotion of netizen. it usually consists of text and images. but at the same time, memes also convey some negative emotions, such as negative comments about women. semeval-2022 task5: multimedia automatic misogyny identification (mami) @xcite focuses on identifying whether meme conveys negative emotions towards women. since the transformer @xcite ) and bert @xcite models were proposed, researchers have begun to work on image and text multi-modality work in recent years, in addition to using one modality such as only image or text. nowadays, for multimodal models, they can be divided into two categories, singlestream model and dual-stream model. in the single-stream model, language information and vision information are fused at the beginning and directly input into the encoder. some representative single-stream models include imagebert @xcite , unicoder vl @xcite , vl-bert @xcite , visualbert @xcite , etc. in the dual-stream model, in addition to the lxmert, we will introduce below, there were vilbert @xcite and unimo @xcite , etc. as for emotion recognition, in previous tasks, there are also emotion classification tasks based on multi-modal graphics and text, such as @xcite used text-cnn and albert to identify the persuasion skills of meme. @xcite used the adversarial learning of sentiment word representations for sentiment analysis. a tree-structured regional cnn-lstm @xcite and dynamic routing in a tree-structured lstm @xcite were used for dimensional sentiment analysis. in previous semeval competitions, @xcite extracted heterogeneous visual representations (i.e., face features, ocr features, and multimodal representations) and explored various multimodal fusion strategies to combine the textual and visual representations. in addition, in multimodal analysis combining images and text, @xcite proposed a parallel channel ensemble model combining bert embedding, bilstm, attention and cnn, and resnet for sentiment analysis of memes. the main difficulty of multi-modality is how to extract the two modalities' features and express the semantics more accurately, which involves the representation of multi-modality, the alignment between multi-modality, and the fusion of multi- my dog is cute little word embedder text feature position emb emb token word embedder text feature position emb emb token fusion transformer encoder sigmoid image embedder image feature emb emb roi features position faster r-cnn modality. for the multi-modal task of text and image, the previous practice is to input the text and image into two different pre-training models for processing the text and image modalities respectively, and then concatenate the output features and predict the emotion. however, this method lacks the processing of the alignment relationship between modalities. the proposed model considers the above three problems in the multi-modality field. inspired by lxmert, we use it as the main framework of our model. we use faster r-cnn @xcite to extract image roi features and their position. for texts, we use bert to extract text embedding. then our system uses lxmert @xcite to deal with the multimodal alignment of text and image. after when two modalities are processed by lxmert, we use the learnable integration mechanism bilinearinteraction layer to fuse these features. the remainder of this paper is organized as follows. in section 2, we described lxmert and our fusion method in detail. the experimental results are presented in section 3. finally, a conclusion is drawn in section 4. ## system overview task a and task b are very similar in model structure except for the output layer. therefore, we in-troduce the model we proposed as a whole. this model can be divided into four parts. they are the embedding layer for image and text preprocessing, the encoder for multi-modal presentation and alignment, the feature fusion layer, and the final output layer. the proposed model is as shown in figure 2 . ## conclusion in this task, we design an image and text multi-modality model based on lxmert for multi-modality representation and alignment, and modality fusion based on the bilinear-interaction layer. compared with the traditional method of stitching two pre-training models for each modality then concatenating two features to predict emotion, this model considers the representation, alignment, and fusion of multi-modality, and achieves better results than the baseline method. at the same time, we found that after adding the bilinear-interaction layer, the performance of the model is better than using only feature concatenate. see