AMS _ ADRN at S em E val-2022 Task 5: A Suitable Image-text Multimodal Joint Modeling Method for Multi-task Misogyny Identification (2022)

Abstract: Women are influential online, especially in image-based social media such as Twitter and Instagram. However, many in the network environment contain gender discrimination and aggressive information, which magnify gender stereotypes and gender inequality. Therefore, the filtering of illegal content such as gender discrimination is essential to maintain a healthy social network environment. In this paper, we describe the system developed by our team for SemEval-2022Task 5: Multimedia Automatic Misogyny Identification. More specifically, we introduce two novel system to analyze these posts: a multimodal multi-task learning architecture that combines Bertweet for text encoding with ResNet-18 for image representation, and a single-flow transformer structure which combines text embeddings from BERT-Embeddings and image embeddings from several different modules such as EfficientNet and ResNet. In this manner, we show that the information behind them can be properly revealed. Our approach achieves good performance on each of the two subtasks of the current competition, ranking 15th for Subtask A (0.746 macro F1-score), 11th for Subtask B (0.706 macro F1-score) while exceeding the official baseline results by high margins.

## introduction women are influential online, especially in imagebased social media such as twitter and instagram : 78 percent of women use social media multiple times a day, compared with 65 percent of men.(elisabetta fersini) however, although the internet has opened new opportunities for women, systematic offline inequality and discrimination are replicated in cyberspace in the form of offensive content against them. the popular communication tool in social media platforms is meme. memetics is essentially an image, which is characterized by the content of the image and the poster cover text introduced by human beings. its main goal is fun or irony. although most of them were created for a joke, in a very short period of time, people began to use them as a form of hatred against women, and then magnified gender stereotypes and gender inequality. women are exposed to gender discrimination and aggressive information in the network environment. therefore, the filtering of illegal content such as gender discrimination is essential to maintain a healthy social network environment. in this paper, we describe the system developed by our team for semeval-2022 task 5: multimedia automatic misogyny identification. more specifically, we introduce two novel systems to analyze these posts: a multimodal multi-task learning architecture that combines bertweet @xcite ) for text encoding with resnet-18 @xcite for image representation, and a single-flow transformer structure which combines text embeddings from bert-embeddings and image embeddings from several different modules such as effi-cientnet @xcite and resnet @xcite . visual and language multimodal learning. some studies explored cross-modal transmission between images and text. some ideas are related to cross-model representation learning @xcite , aiming to generate representations that effectively correlate different sensor modes. recently, several ideas have been proposed to focus on the transformer backbone by improving the pre-training strategy rather than optimizing the backbone structure. videobert @xcite learns the bidirectional joint distribution of visual and linguistic tag sequences, which are respectively from the vector quantization of frame data and the ready-made speech recognition output. visualbert @xcite uses mlm to pretrain specific tasks, which uses the same type id for language & visual input. vl-bert @xcite uses a simple transformer model as the backbone, and uses a mixture of region of interest ( roi ) and words in the image as input. in addition, some ideas have become focused on improving the transformer backbone network. unicoder-vl @xcite uses three pretraining tasks, including mask language modeling (mlm), mask object classification (moc) and visual language matching (vlm), to learn the joint representation of vision and language. vilbert @xcite extends the bert architecture to a multi-modal dual-flow model and processes it in separate flows interacting through the common attention converter layer 3 task description the proposed task, i.e. multimedia automatic misogyny identification (mami) consists in the identification of misogynous memes, taking advantage of both text and images available as source of information. the task will be organized around two main sub-tasks: ## methodology for this task, we have tried a variety of modeling and optimization methods, which are described in detail as follows. ## experiments this section is organized as follows. first we introduce the mami dataset (elisabetta fersini), then we introduce the experimental setup in detail. finally, we show the effectiveness of our proposed method on mami datasets. ## conclusions and future works in this paper, we describe the system developed by our team for semeval-2022 task 5: multimedia automatic misogyny identification. more specifically, we introduce two novel systems to analyze these posts: a multimodal multi-task learning architecture that combines bertweet for text encoding with resnet-18 @xcite for image representation, and a single-flow transformer structure which combines text embeddings from bert-embeddings and image embeddings from several different modules such as efficientnet @xcite and resnet @xcite .in addition, we also use the model fusion and the post-processing method of prediction results, and improve the prediction results. in the future, we will try to get more unlabeled sample data to pre-train the transformer single-flow method more fully and look forward to getting better multimodal methods in twitter scenarios.