B ennett NLP at S em E val-2020 Task 8: Multimodal sentiment classification Using Hybrid Hierarchical Classifier (2020)

Abstract: Memotion analysis is a very crucial and important subject in today’s world that is dominated by social media. This paper presents the results and analysis of the SemEval-2020 Task-8: Memotion analysis by team Kraken that qualified as winners for the task. This involved performing multimodal sentiment analysis on memes commonly posted over social media. The task comprised of 3 subtasks, Task A was to find the overall sentiment of a meme and classify it into positive, negative or neutral, Task B was to classify it into the different types which were namely humour, sarcasm, offensive or motivation where a meme could have more than one category, Task C was to further quantify the classifications achieved in task B. An imbalanced data of 6992 rows was utilized for this which contained images (memes), text (extracted OCR) and their annotations in 17 classes provided by the task organisers. In this paper, the authors proposed a hybrid neural Naïve-Bayes Support Vector Machine and logistic regression to solve a multilevel 17 class classification problem. It achieved the best result in Task B i.e 0.70 F1 score. The authors were ranked third in Task B.

## introduction with the increased availability of internet and connected smart multimedia devices, there has been a rapid increase in the spreading of memes over social media platforms like twitter, facebook, instagram, reddit, etc. memes are images containing text both of which are related to each other with a context. memes usually pertain to an idea, belief, thought, theory, etc. which are usually derived from popular or unpopular culture with factors like place, people, events, and actions affecting it. these memes are an integral source of information about a community @xcite . this is why we require a thorough analysis of these memes. not only that but also due to hate speech which has been the bane of social media since the beginning. these are the platforms where people from different parts of the world come together to interact and share ideas, such platforms encourage the concept of free speech however with this comes the issue of usage of offensive or anti-social content. such content, which is not considered criminal, originates from the intent to harass in the name of criticism, leads to a radicalized person or community. detecting such hate speech and offensive content to curb the spread of negativity and emotional distress using the internet is not only a priority task but a social responsibility. such memes that cause trauma and torment to a directed person or a community needs to be flagged, reported, and taken proper action upon to avoid any unwanted disturbance to the people. this process of filtering online content has been traditionally dominated by manual checking of such content which is not possible considering the current internet activity. however, manual filtering is not only a time-consuming task but a possible mental health hazard too which indicates the need to automate this process. even researchers in @xcite , @xcite were generating memes automatically. memes comprise multiple modalities which are textual and visual. these are context-related modalities and machines need a hybrid approach to leverage both these modalities. this paper discusses the data provided by the task organisers @xcite and the proposed approach towards the problem at hand. it goes through the data pre-processing and augmentation performed on the imbalanced set of data and the application of different machine learning and neural network training on the given data-set which includes recurrent neural network (rnns) @xcite , long short term memory (lstms) @xcite , logistic regression @xcite and neural naive bayes support vector machine (nbsvm) @xcite . ## related work not much solid work has been done to classify memes yet. in the classification of a meme, major work has been done on the image segment @xcite . in this work @xcite the author has mainly focused on images of memes and has tried to take out the features of images such as lines, edges, and contrast or interest points such as blobs or corners. in meme classification there exists a lot of ambiguity because a meme consists of images and text both. by experimentation, it was found that text is the most important part of a meme as images follow a template, and what is written in that template matters more, rather than the template. people often confuse meme classification with sentiment analysis. sentiment analysis is a small part of meme classification. a lot of work has been done in the sentiment analysis segment. the first task of meme classification is sentiment analysis i.e predicting whether a meme is positive, negative or neutral. sentiment analysis is a natural language processing(nlp) task and has many levels. starting from the very beginning sentiment analysis started with a document level classification @xcite moving further to sentence level @xcite and from that it came to phrase level @xcite . if we see there is very little or no work done on sentiment analysis on memes, a lot of work has been done on sentiment analysis on twitter data. a very broad approach of sentiment analysis of twitter data was presented in @xcite . the author here collected the twitter data based on emoticons for example for positive ":) ,:-) " for negative " :( :-(" were used. the author's best approach was multinomial naive bayes. in @xcite author collected data through twitter and tried various classifiers. ## data data is the most important factor in any analytics and processing task. the data was provided to us by the competition organisers @xcite in two parts, the first set was a more of ill-prepared data which contained missing rows, missing annotations, missing images, swapped column values and all of this was revealed while working on it as it was creating a lot of unexpected issues, whereas the second set was an updated and cleaner but a similarly flawed version of the original data. ## proposed work to solve this multi-class classification various approaches were tried, top performer presented here: ## results finally as compared to @xcite our proposed hybrid model achieved 6% higher accuracy. in @xcite the authors achieved best accuracy of 81 percent in contrast, our proposed hybrid model for sentiment analysis of memes achieved 87 percent using neural naive bayes support vector machine(nbsvm). ## conclusion we encourage systematic research and development in the field of memotion analysis. using combined machine learning and training methods i.e quantitative and qualitative, we were able to create a model and empirically validate our results on a dataset of 6992 data points. the results are very encouraging and quite a remarkable feat (secure third position in task b). memotion analysis can be one of the most utilised tools in this era of social media (and in most cases, better than) among all the other highly regarded sentiment analysis tools that are available to us till date.