Codec at S em E val-2022 Task 5: Multi-Modal Multi-Transformer Misogynous Meme Classification Framework (2022)

Abstract: In this paper we describe our work towards building a generic framework for both multi-modal embedding and multi-label binary classification tasks, while participating in task 5 (Multimedia Automatic Misogyny Identification) of SemEval 2022 competition. Since pretraining deep models from scratch is a resource and data hungry task, our approach is based on three main strategies. We combine different state-of-the-art architectures to capture a wide spectrum of semantic signals from the multi-modal input. We employ a multi-task learning scheme to be able to use multiple datasets from the same knowledge domain to help increase the model’s performance. We also use multiple objectives to regularize and fine tune different system components.

## introduction in this paper, we present the system that we have built to participate in semeval 2022 task 5 @xcite , multimedia automatic misogyny identification (mami) challenge. the task is targeted at identification of misogynous memes by basically using the meme's image and pre-extracted english text content as input sources. the task is divided into two main sub-tasks: sub-task a is a binary classification task where a meme should be categorized either as misogynous or not misogynous, sub-task b is a multi-label binary classification task, where the type of misogyny should be recognized among the potential overlapping categories: stereotype, shaming, objectification and violence. generally, meme classification is a challenging task as memes are multi-modal, rely heavily on implicit knowledge, and are subject to human misinterpretation especially among different backgrounds and cultures. we have used transformer @xcite ) based architectures and took a transformer based approach to combine them. transformer based architectures are achieving state-of-the-art performance for natural language processing and computer vision related tasks. there are architectures for language, vision, and language and vision combined. there are also architectures for other modalities however in mami's scope, we are interested in images and text only. pretraining a deep neural network and a transformer based architecture from scratch is a data hungry and computation resources demanding task. especially in a multi-modal domain where input can have multiple image and text modalities. the literature is rich in a variety of architectures which achieve competitive performance in different tasks for different modalities. in our work, we took an approach towards building a framework that would allow us to combine different pretrained architectures. with relatively few epochs and using relatively less compute resources, our goal was to build and train a classifier framework that could harness the power of pretrained architectures as backbones, using relatively limited resources: no multiple gpus for training and constrain the cost to be relatively small, in the range of hundred dollars in total. we have assumed that using as many different backbone architectures which are trained on different tasks for different modalities can allow us to capture a wide spectrum of semantic signals from the input modalities. then we just need to build a classifier to learn the relationship between these signals and the target classes. we have also found and discuss below that there are available text and image datasets for hate speech, sexism and hateful memes which sound related to this task and could be used to augment mami's dataset. eventually, our goal was to combine different datasets in a multi-task classifier. in order to guide the model during training towards the main objective, which is to minimize the classification loss, we have employed different auxiliary objectives. breaking down the whole architecture into components, each component has a sub-target. we have assumed that if we could assure that each component performs well on the subtarget, then the whole system would perform better on the main target. a component producing a clear signal can facilitate the learning of the downstream dependent components. this could increase model performance in terms of either accuracy or convergence speed. so, if we could formulate an objective function for each component, we can linearly combine them with the main objective function. this helps fine tuning and regularizing the components of the system. we have built a generic classification framework and applied it to both sub-tasks a and b. during the evaluation phase of the competition, we have achieved, for sub-task a, a macro-average f1-measure of 0.715, and for sub-task b, a weighted-average f1-measure of 0.698. during post-evaluation phase we have achieved higher score for sub-task a of 0.761. our code is available at https://github.com/ ahmed-mahran/mami2022 . multi-modal frameworks: mmbt @xcite concatenates, into a single sequence, linear projection of resnet @xcite output for image pooled to n different vectors, with bert @xcite tokens embeddings for text. the sequence is fed into a transformer encoder, they call it a bi-transformer, after adding positional embeddings and segment embeddings to distinguish which part is image and which part is text. the architecture is generic enough to use different image encoders. as a variant of how we combine signals from image and text, we extend the mmbt architecture to combine more than two encoders however that wasn't our top performing variant. mmca @xcite combines faster r-cnn @xcite with bert to compute two embedding types for each modality: self-attention embedding to capture intra-modality interactions, and cross-attention embedding to capture both intraand inter-modality interactions. ## experimental setup in addition to mami's dataset, we have used facebook's hateful memes dataset. we have set the tasks configurations as shown in we have split both datasets into 80% train and 20% dev sets using stratified sampling. we have used the data provided at post-evaluation period of the competition as the test set. we have used a batch size of 16 and number of epochs as 15. we have used madgrad @xcite for optimization. learning rate was set to 2 × 10 -4 and we used a learning rate linear scheduler with a warmup period foot_3 . gradients are accumulated every 20 batches so there was a total gradient accumulation steps of: total number of batches/20 × number of epochs. we have set the warmup period to: total gradient accumulation steps/10. for all parameters except biases and layer normalization weights, we have used a weight decay of 5 × 10 -4 . we clip gradients to overall norm of 0.5. our implementation is pytorch based and we have used huggingface transfomers implementation for both bert and detr. we have run our experiments on google colab pro plus using one tesla p100 gpu. we didn't perform any special data preprocessing, just the requirements for each backbone. also, to make experiments quicker, we didn't perform any fine tuning for any of the backbones and we pre-generated and stored each backbone output instead of re-evaluating the same data across epochs and experiments. we used the official accuracy measures to score how the model performs on each task. for the single class tasks, we have used macro-average f1-measure and we called it scorea. in particular, for each class label (i.e. true and false) the corresponding f1-measure will be computed, and the final score will be estimated as the arithmetic mean of the two f1-measures. for the multi class tasks, we have used weighted-average f1-measure and we called it scoreb. in particular, the f1-measure will be computed for each label and then their average will be weighted by support, i.e. the number of true instances for each label. ## conclusion in this paper, we propose a generic framework for both multi-modal embedding and multi-label binary classification tasks. we combine and use as backbones different architectures achieving stateof-the-art in different or similar tasks on different modalities to capture a wide spectrum of semantic signals from the multi-modal input. by employing a multi-task learning scheme, we are able to use multiple datasets from the same knowledge domain and increase the model's performance. in addition to that, we use multiple objectives to regularize and fine tune the system components. we have carried out experiments to verify our ideas and the results show the significance of some of the ideas. as a future work, we need to do more experiments with different backbone architectures and more datasets. we can also try more objectives and regularizations; for instance, we can use our observation from figure 3 to make the decoder output for a positive instance closer to the corresponding class query embedding.