U o R at S em E val-2020 Task 8: G aussian Mixture Modelling ( GMM ) Based Sampling Approach for Multi-modal Memotion Analysis (2020)

Abstract: Memes are widely used on social media. They usually contain multi-modal information such as images and texts, serving as valuable data sources to analyse opinions and sentiment orientations of online communities. The provided memes data often face an imbalanced data problem, that is, some classes or labelled sentiment categories significantly outnumber other classes. This often results in difficulty in applying machine learning techniques where balanced labelled input data are required. In this paper, a Gaussian Mixture Model sampling method is proposed to tackle the problem of class imbalance for the memes sentiment classification task. To utilise both text and image data, a multi-modal CNN-LSTM model is proposed to jointly learn latent features for positive, negative and neutral category predictions. The experiments show that the re-sampling model can slightly improve the accuracy on the trial data of sub-task A of Task 8. The multi-modal CNN-LSTM model can achieve macro F1 score 0.329 on the test set.

## introduction contents of social media are typically multi-modal while traditional natural language processing (nlp) and computer vision methods only process text or image data respectively. in recent times, memes have frequently been used in internet communities and are symbols of modern internet culture @xcite . it typically contains both images and texts to express a certain semantic meaning. sentiment analysis can classify whether a users' opinion is positive based on both the meaning of the image and the metaphor of the text. so it is important to develop multi-modal data process methods to understand the conveyed semantic meaning of the memes. the given data of semeval-2020 task 8 include 6992 human labelled memes image with its text content @xcite . in sub-task a, the requirement is to classify a given meme into positive, negative or neutral category. social media data are typically severely imbalanced. the given data in the sub-task a is also in this case where the instances of the positive class are around 6.59 times more than that in the negative class. class imbalance refers to skewed class distributions in datasets, where the number of samples of one class is significantly greater than the other classes. existing studies shows that class imbalance problems can impose negative effects on the performance of a classification problem @xcite . this is because classifier algorithms are often biased towards the majority classes @xcite . multi-class data are however difficult to balance since the relationship between classes are not straight forward. dealing with multi-class problems such as the memotion analysis problem poses some practical challenges that results in a loss of performance in one data class whilst trying to compensate it in the other class @xcite . @xcite used gaussian mixture models (gmms) as embedding for words vectors, which allows words to have multiple meaning. gmms have been shown to capture a full distribution of images and can also generate realistic image samples although they are not as sharp as the ones from generative adversarial networks @xcite . to handle the data imbalance problem, we therefore propose a gaussian mixture sampling method to balance imbalanced text sequence vectors. ## related work the usage of memes in internet communities has been significantly increased in recent years @xcite . memes are images with texts that are often recreated from prior art content, including painting, operas, cartoon or movie, which typically contains metaphors that help the users express their feelings and emotions @xcite . however, it is found challenging to detect offensive content automatically, since the content is difficult to understand comparing to textual hate speech. moreover, understanding memes is an important way to know public opinions on social media, thus automatic multi-modal content analysis has attracted great attentions in the research communities. @xcite tried to analyse the relevance between memes images and texts on social media in terms of semantic meaning. they found that memes were typically used to emphasis the discussions related to users' sentiment. some other research focused on automatic methods to detect sexiest in multi-modal memes @xcite . recently, many research began to focus on multi-modal methods in deep learning, which can achieve excellent performance on features extraction due to its' nonlinear learning ability. convolutional neural network (cnn) and long short term memory network (lstm) models are popularly used to combine image and text features for sentiment classification. in @xcite , cnn-multi methods were proposed to concatenate image and text features to classify sentiment. moreover, attention based mechanism has been proposed to focus on a specific entity in text and image, aiming to minimise the differences between image and text vectors to train a model @xcite . however, only few approaches have been proposed for handling multi-class imbalanced data. the binarization approach known as one-versus-one (pairwise learning) proposed by @xcite in their work and described by @xcite was adapted in conjunction with a hybrid combined approach to overcome the multi-class imbalance problem faced. @xcite applied static-smote method involving a re-sampling procedure in t steps, where t represents the number of classes. in each iteration, the re-sampling technique selects the minority size class, and duplicates the number of instances of the class in the original dataset. ## data description in the originally released dataset, there are 6988 valid referenced meme images out of 6992 entries, which are used as a training set. also, the released trial set contains 914 (910 valid) different labelled entries, which are used to evaluate performance of models. the test set contains 1878 (1877 valid) unlabelled entries, and produces the results for submission. the original column of "overall_sentiment" has 5 classes, which have been aggregated into 3-class format as sub-task a required. in the training set, there are 4156 meme images belonging to the positive class, 2201 and 631 meme images belonging to the neutral and negative classes respectively making the data highly unbalanced. next we discuss how text and image data have been prepared before modelling. ## conclusions memes analysis can help people better understand opinions on social media. however, memes data in the real world are typically unbalanced. the given memes dataset also demonstrates a significant there are some limitations in our models. due to the memory limits, we only used images in the size of 128 x 128. in future work, larger size of images and other pre-trained text embedding could be used to further improve the performance. we only tested 3-channel rgb images while 1-channel gray-scale images can also be explored. in addition, further research could be focused on applying the proposed gaussian mixture sampling method on image and fusion model.