MMVAE at S em E val-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection (2022)

Abstract: Nowadays, memes have become quite common in day-to-day communications on social media platforms. They appear to be amusing, evoking and attractive to audiences. However, some memes containing malicious contents can be harmful to the targeted group and arouse public anger in the long run. In this paper, we study misogynous meme detection, a shared task in SemEval 2022 - Multimedia Automatic Misogyny Identification (MAMI). The challenge of misogynous meme detection is to co-represent multi-modal features. To tackle with this challenge, we propose a Multi-modal Multi-task Variational AutoEncoder (MMVAE) to learn an effective co-representation of visual and textual features in the latent space, and determine if the meme contains misogynous information and identify its fine-grained categories. Our model achieves 0.723 on sub-task A and 0.634 on sub-task B in terms of F_{1} scores. We carry out comprehensive experiments on our model’s architecture and show that our approach significantly outperforms several strong uni-modal and multi-modal approaches. Our code is released on github.

## introduction with the rapid development of social media, the use of image-based memes has been growing. people use memes for various purposes, such as to express humor @xcite , or to attract greater attention. enabling this, simple websites that allow people to easily create new memes have further seen their proliferation. however, this easy composition has allowed people to easily embed harmful messages within memes, often circumventing more traditional textbased moderation tools @xcite . this paper particularly focuses on misogynous memes (i.e. hatred towards women) -see figure 1 for two examples. as some platforms may choose to limit the sharing of such material, we argue it is vital to build tools that can automatically identify misogynous memes at scale. many prior works have sought to analyze the content of memes. some have looked into meme emotion analysis @xcite , meme ecosystem measurements @xcite and meme auto-generation (vyalla and udandarao, 2020). past efforts have also shed light on hateful meme detection @xcite or offensive meme detection @xcite more generally. this paper builds on these prior works to automatically detect misogynous meme content @xcite . this comes with several key challenges. first, a meme usually comes with both a visual and a textual part. sometimes the standalone image or text is not necessarily hateful or toxic, but when combined together, the semantic meaning becomes harmful. to effectively understand the semantic meaning of a meme, information encoded in both modalities should be considered. second, different memes frequently have the same image, but are embedded with different text (and thus have different and even opposite meanings). this makes reliance on image-hash lists ineffective. it is also common that the image and text of a meme are unrelated, as in figure 1a . in this case, finding an accurate co-representation of both visual and textual features is vital. finally, malicious information contained in a meme can have granular labels and even belong to multiple categories. thus, it is often necessary to devise a more nuanced taxonomy. with the above challenges in mind, we propose mmvae: a pipeline to determine if a meme contains misogynous information, and to identify its fine-grained hateful labels accordingly. specifically, our contributions are as follows: ## background our approach leverages multi-modal learning, pretrained model, variational autoencoder and multitask learning. below, we provide a brief overview of these concepts. ## problem statement the paper strives to build a classifier that can distinguish misogynous vs. non-misogynous memes. we break this down into two sub-tasks: note, the meme's image i has the embedded text on it, and t is transcribed from the meme's embedded text. ## our approach as illustrated in figure 3 , our mmvae model consists of 3 components: (i) the image/text embedding module, (ii) the variational autoencoder module and (iii) the multi-task learning module. the embedding module turns the inputs into unimodal vector representations. after that, the vae module fuses multi-modal representations and generates a co-representation. finally, the multi-task learning module gives the label prediction. in this section, we first introduce each part of our proposed model, and then demonstrate how to put them all together and jointly train the model. ## experimental setup in this section, we first introduce the multimedia automatic misogyny identification (mami) dataset released by the task organizer @xcite and then discuss our experimental settings. ## results and analysis in this section, we present our model's performances on both tasks, and give further analysis on its strengths and weaknesses. ## related work the misogynous meme detection task is novel, but has similarities to other more general hateful meme detection tasks. related datasets has been released on a number of shared hateful meme detection/classification tasks @xcite . most of the prize-winning models adopted visual-andlinguistic pre-trained models. velioglu and rose (2020) utilized pre-trained visualbert @xcite to encode the meme image regions and caption all together. sharif et al. (2021) and zia et al. (2021) leveraged visual and textual pre-trained models to encode the meme image and the embedded text respectively, and then learned multi-modal co-representation through vector concatenation. instead of concatenating the vectors, pramanick et al. (2021) applied self-attention to learn intra-modality semantic alignments. lippe et al. ( 2021 ) used an ensemble of existing multi-modal pre-trained models based on uniter @xcite . @xcite showed that directly applying state-of-theart multi-modal models on hateful meme classification won't get the optimal performance. they used various data pre-processing approaches to get sufficient features, e.g. entity tags, as additional inputs to the pre-trained models. note, there have also been studies of misogyny in uni-modal platforms @xcite @xcite . our work differs from the above. many of these previous works directly leverage uni-modal embeddings produced by pre-trained models. they then build a relatively simple model afterwards, of which the learnt multi-modal features are not integrated. in contrast, we strive to overcome this limitation via the co-representation of both textual and image modalities. moreover, we differ in that we are focusing on misogynous meme detection, rather than the broader topic of hateful meme detection. ## conclusion the spread of hateful memes targeting certain groups has become an important problem on social media platforms. to mitigate the negative consequences brought by misogynous memes, we propose a multi-modal multi-task variational au-toencoder (vae) to identify them and assign them more fine-grained labels. our model consists of three main components: the image/text embed-ding module, the variational autoencoder module, and the multi-task learning module. our model's performance outperforms the state-of-theart unimodal baselines by 22.8% and 16.2%. it effectively learns the co-representation of visual and textual features, and is jointly trained on multiple downstream classification tasks. in our future work, we plan to integrate attention mechanism into our model and carry out more comprehensive statistical analysis on model's results.