Unimodal Intermediate Training for Multimodal Meme Sentiment Classification (2023)

Abstract: Internet Memes remain a challenging form of user-generated content for automated sentiment classification. The availability of labelled memes is a barrier to developing sentiment classifiers of multimodal memes. To address the shortage of labelled memes, we propose to supplement the training of a multimodal meme classifier with unimodal (image-only and text-only) data. In this work, we present a novel variant of supervised intermediate training that uses relatively abundant sentiment-labelled unimodal data. Our results show a statistically significant performance improvement from the incorporation of unimodal text data. Furthermore, we show that the training set of labelled memes can be reduced by 40% without reducing the performance of the downstream model.

## introduction as internet memes (or just "memes") become increasingly popular and commonplace across digital communities worldwide, research interest to extend natural language classification tasks, such as sentiment classification, hate speech detection, and sarcasm detection, to these multimodal units of expression has increased. however, state-of-theart multimodal meme sentiment classifiers significantly underperform contemporary text sentiment classifiers and image sentiment classifiers. without accurate and reliable methods to identify the sentiment of multimodal memes, social media sentiment analysis methods must either ignore or inaccurately infer opinions expressed via memes. as memes continue to be a mainstay in online discourse, our ability to infer the meaning they convey becomes increasingly pertinent @xcite . achieving similar levels of sentiment classification performance on memes as on unimodal content remains a challenge. in addition to its multi-modal nature, multimodal meme classifiers must discern sentiment from culturally specific inputs that comprise brief texts, cultural references, and visual symbolism @xcite . although various approaches have been used to extract information from each modality (text and image) recent works have highlighted that meme classifiers must also recognise the various forms of interactions between these two modalities @xcite @xcite . current approaches to training meme classifiers are dependent on datasets of labelled memes @xcite @xcite @xcite containing sufficient samples to train classifiers to extract relevant features from each modality and relevant cross-modal interactions. relative to the complexity of the task, the current availability of labelled memes still poses a problem, as many current works call for more data @xcite @xcite . worse still, memes are hard to label. the complexity and culture dependence of memes @xcite cause the subjective perception problem @xcite , where varying familiarity and emotional reaction to the contents of a meme from each annotator causes different ground-truth labels. second, memes often contain copyright-protected visual elements taken from other popular media @xcite , raising concerns when publishing datasets. this required @xcite to manually reconstruct each meme in their dataset using licenced images, significantly increasing the annotation effort. furthermore, the visual elements that comprise a given meme often emerge as a sudden trend that rapidly spreads through online communities @xcite , quickly introducing new semantically rich visual symbols into the common meme parlance, which carried little meaning before @xcite . taken together, these characteristics make the labelling of memes particularly challenging and costly. in seeking more data-efficient methods to train meme sentiment classifiers, our work attempts to leverage the relatively abundant unimodal sentiment-labelled data, i.e. sentiment analysis datasets with image-only and text-only samples. we do so using @xcite supplementary training on intermediate labeleddata tasks (stilt) which addresses the low performance often encountered when finetuning pretrained text encoders to data-scarce natural language understanding (nlu) tasks. phang et al.'s stilt approach entails three steps: stilt has been shown to improve the performance of various models in a variety of text-only target tasks @xcite . furthermore, @xcite observed that stilt is particularly effective in target tasks in nlu with smaller datasets, e.g. wic (pilehvar and camacho-collados, 2019) and boolq @xcite . however, they also showed that the performance benefits of this approach are inconsistent and depend on choosing appropriate intermediate tasks for any given target task. in some cases, intermediate training was found to be detrimental to target task performance; which @xcite attributed to differences between the required "syntactic and semantic skills" needed for each intermediate and target task pair. however, stilt has not yet been tested in a configuration in which intermediate and target tasks have different input modalities. although only considering the text or image of a meme in isolation does not convey its entire meaning @xcite , we suspect that unimodal sentiment data may help incorporate skills relevant to discern the sentiment of memes. by proposing a novel variant of stilt that uses unimodal sentiment analysis data as an intermediate task in text encoder pretraining image encoder pretraining meme sentiment classification text encoder pretraining target task (finetuning) supervised intermediate task (̸ = target task) meme sentiment classification image sentiment classification image encoder pretraining text encoder pretraining meme sentiment classification text sentiment classification image encoder pretraining text encoder pretraining baseline (meme classifier finetuning) phang et al.'s (2019) stilt image-stilt text-stilt task modality visual textual multimodal figure 1: training tasks in baseline, phang et al.'s (2019) stilt, and our proposed image-stilt and text-stilt approaches. rq1: does supplementing the training of a multimodal meme classifier with unimodal sentiment data significantly improve its performance? we separately tested our proposed approach with image-only and text-only 3-class sentiment data (creating image-stilt and text-stilt, respectively) as illustrated in figure 1 ). if either proves effective, we additionally answer: rq2: with unimodal stilt, to what extent can we reduce the amount of labelled memes whilst preserving the performance of a meme sentiment classifier? ## methodology to address our research questions, we chose the 3-class sentiment polarity of multimodal memes as our target task as defined by @xcite for our chosen dataset. our experimental approach revolves around comparing the performance of a multimodal classifier trained only on memes (our baseline) and those trained first on unimodal image or text data (our image-stilt and text-stilt models, respectively) before being trained on memes. these models are architecturally identical to each other, all trained in the memotion 2.0 training set and tested against the memotion 2.0 testing set to isolate the effect of unimodal intermediate training on meme sentiment classification performance. the results of the performance of the model are measured using the weighted f1-score, as defined by the authors of the selected meme dataset @xcite . a detailed description of this metric is available in appendix b. ## limitations and future works to generate comparable results between baseline and text-stilt, we kept many hyperparameters constant. additional work would be required to determine the maximum achievable performance of text-stilt on the chosen task. despite the efficacy of text-stilt over image-stilt, these results do not suggest that only the text modality is significant in classifying multimodal memes. previous works have performed modality ablation studies in this problem space @xcite @xcite with multimodal architectures remain-ing the apparent state of the practise. all models in this work are similarly multimodal. in the future, we plan to reformulate image-stilt with respect to the approach and data used to isolate the cause of its non-performance on the downstream task. furthermore, we did not test text-stilt on classifiers that represent the image modality of a meme in textual forms, as others did @xcite . notwithstanding our results, text-stilt may not benefit all multimodal meme classifiers. phang et al. (2019) showed that stilt offers varying degrees of benefit depending on the encoders chosen. future work is needed to verify if these observations hold across the wide range of pretrained encoders commonly used in meme classifiers. in particular, some modifications to unimodal stilts are needed to be applied to single-stream multimodal encoders, as those used in other works. furthermore, pruksachatkun et al. (2020) showed that intermediate training benefits various text-only tasks differently. we have yet to identify other meme classification tasks that would benefit from unimodal stilts. thus, we plan to conduct more extensive experimentation to validate the effectiveness of text-stilt on other meme classification tasks, e.g. pairing hate-speech detection in text @xcite as an intermediate task for hateful meme detection @xcite . ## conclusion in this work, we addressed the challenge of training multimodal meme sentiment classifiers on a limited number of labelled memes by incorporating unimodal sentiment analysis data. we did so by proposing the first instance of stilt that applies unimodal intermediate tasks to a multimodal target task. specifically, we tested image-only and text-only sentiment classification as intermediate tasks in training a meme sentiment classifier. we showed that this approach worked -unimodal text improved meme classification performance to a statistically significant degree. this novel approach allowed us to train a meme classifier that outperforms meme-only finetuning with only 60% as many labelled meme samples. as possible explanations for our observations, we discuss apparent similarities and differences in the roles of image and text modalities between unimodal and multimodal sentiment analysis tasks. layers of reducing sizes @xcite @xcite , dense i and dense t for the image and text modalities, respectively. then, softmax is applied on the output of each stack is to generate a weighted score for each modality. per @xcite : we added a dropout and normalisation step onto the fused multimodal representation, f m m : the predicted logits of each class is given by passing f m m a dense network of gelu-activated layers with sizes @xcite @xcite : the model is fitted by minimising the mean multiclass cross entropy loss per pytorch's definition: where x n,yn is the logits for each class and y n is the target label of a given sample n of total n samples in a minibatch; c is the class in [negative, neutral and positive] and c is the number of classes. the loss for each sample is weighted by: where n 0 , n 1 , n 2 are the number of training samples labelled with negative, neutral and positive sentiment, respectively.