DD - TIG at S em E val-2022 Task 5: Investigating the Relationships Between Multimodal and Unimodal Information in Misogynous Memes Detection and Classification (2022)

Abstract: This paper describes our submission for task 5 Multimedia Automatic Misogyny Identification (MAMI) at SemEval-2022. The task is designed to detect and classify misogynous memes. To utilize both textual and visual information presented in a meme, we investigate several of the most recent visual language transformer-based multimodal models and choose ERNIE-ViL-Large as our base model. For subtask A, with observations of models’ overfitting on unimodal patterns, strategies are proposed to mitigate problems of biased words and template memes. For subtask B, we transform this multi-label problem into a multi-class one and experiment with oversampling and complementary techniques. Our approach places 2nd for subtask A and 5th for subtask B in this competition.

## introduction online misogynous speech has been a worldwide phenomenon spread widely across social media platforms where women are increasingly subjected to offensive content. it has been shown that women are twice as likely as men to encounter online sexual harassment and gender-based violence @xcite . the problem with misogyny detection is that it requires context and external knowledge to understand online speech, which sometimes can be very short and contain subtle meaning @xcite . since memes are getting popular as communication tools on social media platforms, misogynous memes have the potential to affect everyone in our society. automatic multimodal internet memes identification becomes a new challenging type of misogyny detection task that can only be solved by joint reasoning and understanding of visual and textual information @xcite . the proposed task multimedia automatic misogyny identification (mami) @xcite at semeval-2022 requires participants to identify misogynous memes (subtask a) and classify them as certain overlapping categories: stereotype, shaming, objectification, and violence @xcite . this paper describes the system developed by the dd-tig team for semeval-2022 task 5 mami. this work contributes to the following: for subtask a, solutions to biased words and template memes are proposed to mitigate the effects of overfitting in unimodal information. we also utilize ensemble learning and external knowledge source like perspective api to boost the performance of our system. for subtask b, we transform the multilabel classification problem into a multi-class classification problem and reach a better result with oversampling and complementary strategy. ## experimental setup in our baseline approaches, visualbert-large, oscar-large, and uniter-large provided by villo are trained for 5 epochs with a batch size of 16. ernie-vil-large models provided by the original author are trained for 5000 steps with a batch size of 8. in our work, not much time was spent on hyperparameter optimization, and since we notice that there is not much difference when training models with varying hyperparameter settings and we focus more on other strategies. the hyperparameters for finetuning ernie-vil-large are presented in appendix b. ## conclusion in this paper, we have presented our work on multimedia automatic misogyny identification (mami) at semeval-2022. mainstream vision-language models are applied on misogynous memes dataset in the baseline approach. for subtask a, to better utilize multimodel information and unimodal information, we propose solutions to mitigate the effects of biased words and templates memes. ensemble learning and external knowledge source like per-spective api are used to enhance the performance of our system. for subtask b, training with oversampling strategy, we use a multi-class model to solve this multi-label problem and gain improvement from our subtask a model. in short, this task could never be solved easily since it relies heavily on the context, external knowledge, relations between modalities.