taochen at S em E val-2022 Task 5: Multimodal Multitask Learning and Ensemble Learning (2022)

Abstract: We present a multi-modal deep learning system for the Multimedia Automatic Misogyny Identification (MAMI) challenge, a SemEval task of identifying and classifying misogynistic messages in online memes. We adapt multi-task learning for the multimodal subtasks of the MAMI challenge to transfer knowledge among the correlated subtasks. We also leverage on ensemble learning for synergistic integration of models individually trained for the subtasks. We finally discuss about errors of the system to provide useful insights for future work.

## introduction multimodal machine learning processes data from different modalities (e.g. visual, auditory, lingual) to infer their combined meaning. this topic has seen tremendous development, encompassing visual question answering (vqa) @xcite , image captioning @xcite , multimodal classification and beyond. semeval 2022 task 5 "multimedia automatic misogyny identification" (mami) @xcite also requires multimodal machine learning to analyze both visual and textual information from memes (image, caption) in order to identify and classify misogynistic memes. the mami challenge has the following two subtasks: as introduced by a multimodal research survey @xcite , data representation, fusion, and co-learning are the primary challenges * corresponding author in multimodal classification problems. to address those challenges, we explore the following relevant topics: data augmentation, multimodal pre-training, multi-task learning and ensemble learning. data augmentation helps enrich under-represented features in data. multimodal pre-trained models may represent multimodal data like those of the mami challenge better than unimodal pre-trained models. we also adapt multi-task learning to transfer knowledge in training data among related tasks. as the misogyny sub-categories are correlated to each other, this approach is especially useful in subtask b. our final predictions are determined by majority voting among the top performing models. this paper is structured as follows: in section 2, we describe the task description and dataset. section 3 explains in details the models and methods we incorporate into the final system architecture. section 4 discusses the evaluation results of the models and methods, including multi-task learning and ensemble learning. we conduct error analysis in section 5 and conclude our findings in section 6. ## task description and dataset semeval 2022 task 5 "multimedia automatic misogyny identification" (mami) @xcite is a classification task that aims at identifying and classifying misogynistic memes in social media. while most memes are funny and harmless, some deliver misogynistic content and have strong, negative influence due to their high speed of spread. such memes need to be detected and removed from online sites to avoid gender-related hate. in particular, the mami task targets memes, each of which is essentially an image characterized by a pictorial content with an overlaying text a posteriori introduced by human, thus a multi-modal (image+text) analytics task. the mami task has two subtasks. in subtask a, participants must classify memes into misogynistic and non-misogynistic. the evaluation metric is macro-average f1 score. subtask b is to identify the type of misogyny from the possibly overlapping categories: shaming, stereotype, objectification, and violence if the meme is misogynistic. the evaluation metric is weighted-average f1-measure, where the f1 scores of the four categories are averaged with weights by support, i.e. the number of true instances for each label. the training dataset contains 10,000 memes with english captions, assembled from social media platforms. another 1,000 memes are given as the test dataset. besides memes in jpeg file format, the task also provides transcriptions of memes captions in a separate text file. parts of the data are collected and evaluated as described in @xcite . ## methods we present a system that utilizes a multi-task learning method to deal with dependencies between the subtasks, fills in the lacking parts of the training dataset by using data augmentation methods, and combines the outputs of multiple base models with ensemble methods. ## results we show the evaluation results of the baseline models, multi-task learning and ensemble methods, and then based on the results, present the overall architecture of the final system we used for our official submission of the mami challenge in section 4.4. ## conclusion in this paper, we presented our system for the multimedia automatic misogyny identification (mami) task in semeval 2022. we augmented the data with visual transformation techniques and extracted features from memes by using multimodal baseline models. we further enhanced the system by adapting multi-task learning and ensemble learning methods. we leave issues such as incorporating external information about entities, frame layout in memes, gender-related text data augmentation and cross-subtask dependency in multi-task learning as future works.