SRCB at S em E val-2022 Task 5: Pretraining Based Image to Text Late Sequential Fusion System for Multimodal Misogynous Meme Identification (2022)

Abstract: Online misogyny meme detection is an image/text multimodal classification task, the complicated relation of image and text challenges the intelligent system’s modality fusion learning capability. In this paper, we investigate the single-stream UNITER and dual-stream CLIP multimodal pretrained models on their capability to handle strong and weakly correlated image/text pairs. The XGBoost classifier with image features extracted by the CLIP model has the highest performance and being robust on domain shift. Based on this, we propose the PBR system, an ensemble system of Pretraining models, Boosting method and Rule-based adjustment, text information is fused into the system using our late sequential fusion scheme. Our system ranks 1st place on both sub-task A and sub-task B of the SemEval-2022 Task 5 Multimedia Automatic Misogyny Identification, with 0.834/0.731 macro F1 scores for sub-task A/B correspondingly.

## introduction much of the real world's information comes in multimodality, a combination of images, texts, audios and so on. multimodal understanding aims to utilize different modal of information to improve the overall system recognition intelligence or robustness @xcite , which plays a key foundation role in cognitive ai and embodied ai. with transfer learning by large deep models and colossal corpus achieving remarkable success in vision and language domain, there is a rising interest in combining both sides' advances to push the multimodality understanding further @xcite @xcite @xcite @xcite . we will limit the discussion scope of multimodal to vision and language in this paper. there are two kinds of representative architecture of multimodal learning models, singlestream models and dual-stream models. singlestream model fuses the image and text data at an early stage, and then feed into the model. dualstream models design separated structure as image encoder and text encoder, and a further module is stacked on top of the unimodel encoders for crossmodal learning objectives @xcite @xcite . usually per-unimodal objectives and multimodal objectives are designed to ensure that the model learns unimodal and crossmodal knowledge, like masked image prediction, masked token prediction, and text-image pairing @xcite . two kinds of data distributions are explored for the large-scale pretraining, strongly paired data @xcite @xcite and weakly paired data @xcite . the different distributions would directly affect the correlations learned by the model, yet each pretraining corpus only falls in one pattern. the semeval-2022 task 5 @xcite ) multimedia automatic misogyny identification (mami) is a multimodal classification task in english. it targets the identification of misogynous memes (characterized by a pictorial content with an overlaying text a posteriori introduced by human), using the image and text from the meme as input data. it has two sub-tasks: sub-task a: 2fold classification, to identify whether a meme is misogynous or not; sub-task b: 4-fold fine-grained classification, to further recognize the misogynous meme among potential overlapping categories of stereotype, shaming, objectification and violence. the relationship of the mami paired meme image/text data can vary from highly correlated to weakly correlated or not correlated at all. the semantic logical relationship between the meme's image and text can be: 1) align with each other, containing the same semantic, 2) independent but connected to form a complete semantic, 3) irrelevant with each other, only one modality decides the meme's semantic (refer to the appendix for illustrations). in summary this task demands both understanding the image and text, as well as setting up the correct semantic logic between the image/text modalities. in this paper, we want to investigate either the multimodal pretrained models of different design architecture are capable of handling such complex relationship between vision and language, and whether the pretrain-and-fine-tuning paradigm is advantageous over the feature-extraction-andmachine-learning-classification paradigm. we choose two strong baseline pretrained models, uniter @xcite as the single-stream model, and clip @xcite as the dualstream model, and fine-tuning with cross-entropy softmax is compared against the widely adopted xgboost @xcite classifier. domain shift is discovered between the train (dev) and test dataset, and the above two paradigm is explored for both in-domain situation and domain shift situation. an adversarial discrimination loss @xcite is added to the fine-tuning deep neural network for domain shift, while the xg-boost classifier is tuned with its hyper-parameters. our results show that 1) for both pretrained models, multimodal fine-tuning performs better than unimodal. the clip dual-stream model performs sightly better than the uniter singlestream model on in-domain data, given the much greater pretraining corpus clip has than uniter. on data with domain shift, the clip fine-tuning is much more stable than the uniter model, but both models suffer from great performance degradation. 2) the performance of feature-extractionand-machine-learning-classification by xgboost classifier is no weaker than that of fine-tuning on top of pretrained models, the xgboost classifier utilizing only image features from clip hits best performance on the test dataset among all modality combinations, plus that the xgboost is cheaper to train. 3) for domain shift, the xgboost classifier is more robust, the domain adversarial loss for finetuning brings a small rise, but still falling behind the xgboost classifier. our final winning system is a combination of machine learning and deep neural network, by pretraining models, boosting method and rule-based adjustment, which we name pbr, based on the xgboost classifier of clip image features, and a late sequential fusion of multimodal/text information into the classifier's prediction, followed by rule-based adjustment. the details will be stated in section 2. our system gets 0.834 macro f1 score on sub-task a and 0.731 macro f1 score on sub-task b, ranking 1st place in both the tasks in the leaderboard. ## conclusion in this paper, we investigated the single-steam model uniter and dual-stream model clip's performance on the downstream multimodal classification task, and compared the pretrain-and-finetuning paradigm over the feature-extraction-andmachine-learning-classification paradigm. the experiment results show that the clip per-forms better than uniter on the mami task, and is more robust on domain shift. the uniter unimodal fine-tuning results are significantly worse than the unimodal pretrain model benchmark, suggesting its weakness in handling the complicated semantic logical relationship in the mami task. wheras the structure of clip image feature extraction and xgboost classificatin achieves the highest baseline performance. we proposed the late sequential fusion scheme to fuse text information into our system pbr, and exploited extra data and mutual adjustment of the two sub-tasks to further improve the system performance. our system ranks 1st place in both the sub-tasks in the leaderboard of the semeval-2022 task 5 mami. (a) (b) (c) (d) figure 8: text analysis of train dataset and test dataset. (a) and (b) corresponds to unigram features. (c) and (d) corresponds to bigram features of train and test dataset.