Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance (2022)

Abstract: We propose our solution to the multimodal semantic role labeling task from the CONSTRAINT’22 workshop. The task aims at classifying entities in memes into classes such as “hero” and “villain”. We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we perform qualitative analysis on the representations of the entities.

## introduction social media memes can be defined as "pieces of culture, typically jokes, which gain influence through online transmission" @xcite . more specifically, memes are visual templates usually associated with a textual caption. analysing memes involves many unique challenges that differ from classical multimodal tasks such as image captioning and visual question answering. while unimodal models can often perform well on multimodal datasets @xcite , memes involve a lot of entanglement -stylistic or semantic -between the two modalities, such as the caption contradicting the image. this makes memes intrinsically multimodal. furthermore, pragmatics -the context's contribution to meaning -plays a key role in the interpretation of memes. in particular, phenomenons such as irony are challenging to detect. even human annotators have difficulties in interpreting a meme correctly without knowledge of the community in which the meme was shared. in this paper, we tackle the shared task on multimodal semantic role labeling of the con-straint'22 workshop @xcite . given a (meme, entity) pair, 1 the goal is to classify the entity's role in the meme into one of four classes (hero, villain, victim or other) from the perspective of the author of the meme. the multimodality of the problem stems from the meme, which is given as an (image, ocr) pair, where ocr (for optical character recognition) is the caption extracted from the image. the dataset covers one language, english, and two domains, covid-19 and us politics. figure 1 shows a sample from the training set. understanding memes involves a lot of commonsense and cultural knowledge on the political stance of the entities. thus, it requires models pre-trained on a large amount of data, capable of recognising key entities such as political figures in both modalities, and of inferring their relationship, their role and the public opinion of a community on them. to evaluate the task's difficulty, we manually annotate a set of samples. with 5 annotators, we reach an average macro-f 1 of 0.65 (see details in appendix a), less than 10 points above the best system submitted to the shared task. we propose systems relying on several multimodal (vision-language) pre-trained models: one for all @xcite , clip @xcite and @xcite . we use these models as encoders to extract multimodal meme representations. these encoders are introduced in section 3. we then design several neural network classifiers to handle these representations in a task-specific fashion. these classifiers are presented in section 4.1. the constraint'22 dataset is characterised by a large class imbalance, with the most frequent class gathering 78% of the samples in the train set, while the least frequent one is conveyed by less than 3% of the samples. however, the challenge is evaluated using a macro-f 1 metric and calls for balanced performances across all classes. to handle this discrepancy, we developed several subples, thus considering all entities of a meme independently during training and inference. sampling strategies that we present in section 4.2. our best results are obtained by ensembling predictions from all of our models, using various ensembling methods. the details of the ensembling methods are given in section 4.3. finally, we present our performance in section 5 along with a qualitative analysis of our models. we highlight the limitations of the dataset, task and methods in section 6. to summarise, our whole architecture is built on freely available pre-trained models. we only fine-tune these models for the multimodal semantic role labeling task. this makes computational training cost particularly low. our system can be characterised by: ## related work multimodal semantic role detection in memes is a relatively unique task, compared to other languageimage multimodal task such as object classification and entity action detection, it requires a lot more contextual and cultural background. in this section, we list some related problems before introducing tools to tackle the task at hand in the next section. in recent years, social media platforms have seen a wave of multimodal data in diverse media types. this attracted the interest of researchers to combine modalities to solve various tasks with joint representations, where the model's encoder takes all the modalities as input, or separated representations, where all modalities are encoded separately @xcite . in the constraint'22 challenge, we tackle multimodal semantic role labeling (srl). srl is originally a natural language processing (nlp) task which consists in labeling words in a sentence with different semantics roles to determine who did what to whom, when and where @xcite ; these roles are also known as thematic relations. it was extended to the computer vision domain through visual srl. visual srl benchmarks focus on situation recognition in images (silberer and pinkal, 2018; pratt et al., 2020); these tasks heavily rely on object detection systems for visual groundings @xcite . this differs from the methods we need to implement for the shared task, where the entities do not necessarily appear in the image. moreover, in our case, the semantic role is taken from the point of view of a political argumentative: the perception of the entity by the author of the meme. this involves completely different features compared to labeling the thematic relations of the entity; in particular, cultural and contextual knowledge on the background of the meme. another similar task is multimodal named entity recognition, which aims at identifying and classifying named entities in texts and images. it requires more in-domain knowledge compared to multimodal srl; but most multimodal ner datasets are text-centric, with the image being an additional feature for the text-based prediction @xcite , while our task is more symmetrical or even image-centric. finally, many shared task on memes have been proposed in recent years, with a large variety of tasks: emotion classification (e.g. memotion task at semeval 2020 @xcite ; hateful meme detection (e.g. the hateful meme challenge @xcite event clustering (e.g. dankmemes at evalita 2020 @xcite ); more fine-grained hateful content analysis (fine-grained hateful memes detection mathias et al., 2021, aiming at classifying the target attacked by the meme and the type of attack); or and detection of persuasion techniques (e.g. semeval 2021 task 6, @xcite . els with good performances on popular datasets. these models are multimodal transformers, that we use to encode image and caption's ocr into a common latent space. while transformers were originally developed for natural language processing @xcite , they subsequently became ubiquitous in computer vision models as well @xcite . to process an image, it is first cut into a sequence of p × p × c patches. these patches are then projected into the transformer input dimension, either using a single linear layer, or using a full-fledged cnn architecture. the output of a transformer has the same length as its input. we call this length n ; it is the number of patches in the image, the number of tokens in the ocr, or the sum of the two for multimodal transformers. thereafter, we refer to an encoded meme image i and ocr o as enc full (o, i) ∈ r n ×d . this output can be further pooled into a fixed-size representation enc pool (o, i) ∈ r d . we now describe what models are behind these encoder functions. ## models we now describe how we use the encoded text and images for semantic role labeling. ## discussion the multimodal aspect is crucial in this task. when looking at entity names, only 15% have an exact surface form match in the caption's ocr; moreover, the ocr is often incomplete or noisy (see example in figure 1 with the "exit" sign popping in the middle of the caption). thus, using only the text is far from sufficient. on the other hand, recognising the entities in the image of the meme is not an easy task. as stated in the introduction, the image and the text are often not directly related. moreover, the image often contains elements not seen in common image datasets; for example, meme creators often perform montages like swapping faces and objects. overall, a lot of commonsense and cultural knowledge is needed for the model to understand what the meme is about. the absence of contextual information also makes the task difficult for humans. to evaluate the difficulty of the task, we performed human annotation of a sample of 100 (image, entity) pairs with five annotators. details of annotation process can be found in appendix a. the average pairwise cohen's κ @xcite , used to measure the inter-annotator agreement, is 0.47. it indicates a "moderate" agreement according to @xcite . however, it also shows that less than one third of the annotations are reliable @xcite . moreover, the macro-f 1 scores are relatively low: the average is 0.65 and the maximum 0.69. having metadata such as source website and date of publication of the meme would help human and algorithmic annotators alike. finally, from a real-world point of view, this task is not entirely complete: the ocr and the list of entities are already provided in the dataset, and we only have to perform the classification. in a reallife setting, we would create a multi-task system jointly extracting the caption, detecting entities and classifying them; the three tasks complementing each other. ## conclusion in this work, we propose several systems to solve the task of classifying entity roles in memes. we focus on comparing classification models -mlp, attention and seq2seq systems -on top of pre-trained multimodal encoder: clip, visualbert and ofa. our best standalone system uses the clip encoder with mlp classifier, but our best score is obtained using ensembling of a large number of models. we also compare several sampling strategies to deal with the class imbalance issue, proposing dynamic sampling methods that outperform the standard uniform ("macro") sampling. as a preliminary future work, more or less straightforward processing can be performed on the dataset, at the entity-level (using an entity linker to resolve surface forms to entity identifiers, e.g. merging entities "us" and "united states" together); at the ocr-level (performing lexical normalization @xcite to deal with ocr errors and meme-specific syntax); and at the image-level (removing the text from the image, for a less noisy image embedding). to improve the model, entity representation is key. we wish to train global entity embedding, shared across the whole dataset, and contextualised entity embeddings, aligning the entity's vector representation in the image and in the ocr of the meme (when there is an explicit mention of it). these scores show the difficulty of the task for a human. for comparison, the best score during the challenge was 0.58, still considerably lower than the human best score. to measure the inter-annotator agreement, we compute the average pair-wise cohen's κ @xcite . it is similar to measuring the percentage of agreement, but taking into account the possibility of the agreement between two annotators to occur by chance for each annotated sample. the average cohen's κ is 0.47, indicating a "moderate" agreement according to @xcite . however, it also indicates that less than one third of the annotations are reliable @xcite .