SIS @ IIITH at S em E val-2020 Task 8: An Overview of Simple Text Classification Methods for Meme Analysis (2020)

Abstract: Memes are steadily taking over the feeds of the public on social media. There is always the threat of malicious users on the internet posting offensive content, even through memes. Hence, the automatic detection of offensive images/memes is imperative along with detection of offensive text. However, this is a much more complex task as it involves both visual cues as well as language understanding and cultural/context knowledge. This paper describes our approach to the task of SemEval-2020 Task 8: Memotion Analysis. We chose to participate only in Task A which dealt with Sentiment Classification, which we formulated as a text classification problem. Through our experiments, we explored multiple training models to evaluate the performance of simple text classification algorithms on the raw text obtained after running OCR on meme images. Our submitted model achieved an accuracy of 72.69% and exceeded the existing baseline’s Macro F1 score by 8% on the official test dataset. Apart from describing our official submission, we shall elucidate how different classification models respond to this task.

## introduction sentiment analysis of online social media has become an increasingly hot topic for researchers and has a wide range of applications from e-commerce, analyzing movie reviews, analysis of current trends, campaigns, etc. understanding the sentiment bound to a phrase or sentence helps us discern the opinion of a person and a majority of this information is found on online social media such as instagram, facebook and twitter. in concurrence with this, memes shared online are becoming an increasingly popular method for people to express themselves and their opinions, often in the form of a seemingly humorous visual image, on social media. according to merriam-webster online (2020), a meme is an idea, behavior, or style that spreads by means of imitation from person to person within a culture and often carries symbolic meaning representing a particular phenomenon or theme. they are often shared on social media in the form of an image or even a video. this phenomenon poses a great challenge to conventional nlp systems, which are generally purely text based. a hybrid approach is needed to tackle the problem of analysing the sentiment of a meme. it is therefore becoming exceedingly apparent for both the fields of nlp and computer vision to rise up to the challenge of adopting multi modal approaches as memes, videos, and other multi modal content are becoming increasingly prevalent online. through our submission, we aimed to perceive how basic machine learning models performed in this problem space. our text classification model achieved a macro-f1 score of 29.65% on the task leaderboard, which exceeded the existing baseline by 8% and ranked 29th among the 566 teams registered which is significant given the simplicity of the model. ## related work memes are steadily influencing social media and taking over the feeds of the public. @xcite brought attention to the fact that people use memes to express themselves, and in making them, decide to proclaim their stance on a certain social issue, be it in support or in rejection of the issue. foster (2014) explored how memes were used in the political context to influence facebook users during the 2012 presidential elections. there is always the threat of malicious users on the internet posting offensive content, even through memes. a user may come across such distasteful memes and may flag it down, however, it is impossible for them to catch all offensive material at the scale at which multimodal social media grows everyday. hence, the automatic detection of offensive images/memes is imperative along with detection of offensive text. however, this is a much more complex task as it involves both visual cues as well as language understanding and cultural/context knowledge. work has been done in the automatic generation of memes (peirson and tolunay, 2018) but not much has been done with respect to their analysis and classification of finer aspects like the type and extent of humor and offensiveness incorporated. @xcite aims to detect astroturfing in memes on microblog streams via supervised learning based on features extracted from the topology of the diffusion networks, sentiment analysis, and crowdsourced annotations. @xcite proposes a fully automatic, unsupervised, and scalable for real-time detection of memes in streaming data by exploring the idea of pre-clustering. @xcite attempts to analyse internet memes associated with racism and implements critical discourse analysis in combination with multimodal discourse analysis but there is still a lot of work that needs to be done in the automatic and computational categorization and detection of abusive content in memes. ## shared task description the memotion analysis task was divided into three sub-tasks: sentiment classification, humor classification, and the scales of semantic classes. the first task (task a) was to classify a given internet meme as a positive, negative or neutral meme. task b was to identify the type of humor expressed by an internet meme. in this task, a meme can be labelled as having more than one category. to work with this, one may have to deal with the consequences of treating the multi labelled data, represented as whole, singular labels of various other labels. questions of how to categorize memes found in the test data with different sets of labels from those found in the training data would arise. the third task (task c) is to quantify the extent to which a particular effect is being expressed. the semantic classes to quantify this extent were given as sarcastic, humorous and offensive and were further to be labelled as "not(0)", "slightly(1)", "mildly(2)", and "very(3)" depending on the extent of the expression. more details can be found in the task description paper @xcite . ## data the memotion analysis task released 8k annotated memes (of which 7k were released for training and the remaining 1k was used for testing) with human-annotated tags, namely sentiment, and type of humor (i.e. sarcastic, humorous, or offensive). the received dataset was a compilation of links to the images and the text results for ocr done on the image. the training data was additionally labelled for how humorous/sarcastic/offensive/motivational a given image was, along with its overall expressed sentiment. the composition of the labelled data was as follows. for task a, we have formulated the problem as a text classification problem, splitting the given training data into train/dev data as 80% / 20% respectively. the data given was originally categorized as "negative", "very negative", "positive", "very positive", and "neutral". this was to be narrowed down further and classified into 3 classes: "positive", "neutral" and "negative". ## methodology in this section, we discuss the ten variations of training models, and their corresponding feature sets, that were used to evaluate the performance of simple text classification algorithms in the given problem space. ## results and analysis from the results given in it makes sense that the results received when comparing the accuracy of word-level and characterlevel features are similar as the data is mostly concise monolingual text(the given memes predominantly used english). character-level features generally tend to show promising improvement when the data is bilingual/code-mixed. systems like sub-word level compositions with lstms @xcite to capture sentiment at morpheme level are effective to a certain degree, but are limited by their dependence on abundant data that is beyond which was provided for the task. given that the amount of data is scarce, such deep learning models or use of transformers would fall short as they require an abundance of data to train accurately. hyperparameter tuning worked best for unigram or at most bigram parameters. this may have to do with the length of the input text. models that perform better when the given text input is short in length would perform better when analysing memes due to the inherent structure of a meme. the structure of a meme is such that creators shorten the amount of textual data provided to the extent possible as much of the context for the meme is generally derived from visual input from the image and cultural/general knowledge. ## conclusion and future work in this paper, we explored and elaborated on various training models for simple text classification algorithms. it is a great starting point to evaluate the performance of basic machine learning models in this problem space. the idea of analysing the sentiment of a meme is a newly blossoming research field with limited work done in the past. it therefore holds a lot of promise. as much of the context for the meme is generally derived from visual input from the image and general knowledge, future work may consider the possibility of looking into a hybrid model that could process the image as well as the text to give more context to the ocr outputted data. it is crucial to keep in mind limitations such as the limited text associated with memes. we may also look into methods to procure the cultural/general knowledge needed to understand the meme as well.