Gundapusunil at S em E val-2020 Task 8: Multimodal Memotion Analysis (2020)

Abstract: Recent technological advancements in the Internet and Social media usage have resulted in the evolution of faster and efficient platforms of communication. These platforms include visual, textual and speech mediums and have brought a unique social phenomenon called Internet memes. Internet memes are in the form of images with witty, catchy, or sarcastic text descriptions. In this paper, we present a multi-modal sentiment analysis system using deep neural networks combining Computer Vision and Natural Language Processing. Our aim is different than the normal sentiment analysis goal of predicting whether a text expresses positive or negative sentiment; instead, we aim to classify the Internet meme as a positive, negative, or neutral, identify the type of humor expressed and quantify the extent to which a particular effect is being expressed. Our system has been developed using CNN and LSTM and outperformed the baseline score.

## introduction according to wikipedia article on internet memes, "a meme is an idea, behavior, or style that spreads from person to person within a culture often with the aim of conveying a particular phenomenon, theme, or meaning represented by the meme". meme is not only about the humorous picture, the internet culture, or the sentiment that passes along, but also about the richness and distinctiveness of its language: it is often greatly structured with unusual written style. the internet memes often include superimposed text description with broken grammars and spellings variations. nowadays, internet memes come in almost every form of media, with modish formats continuously expanding. initially, they work as a medium for humor to be shared, using cultural themes. however, they can also be manipulated to further political ideals, company promotions, and social media marketing. in this paper, we present a deep learning based multimodal approach for semeval 2020: task 8 on memotion analysis -the visuo-lingual metaphor! @xcite . this task has three subtasks which describe as follows: all our models are trained using only the corpus supplied by memotion analysis organizers. the evaluation metric for subtask a is macro f1 score and macro f1 score for each of the subtasks, and then average for subtask b and c. primarily, we started experiments with traditional machine learning algorithms foot_1 like support vector machine (svm) and logistic regression (lr). based on the findings from them, we moved to deep learning models foot_2 like long short-term memory (lstm), cnn in search of the better model. in the next section, we summarize the related work. section 3 gives details about the dataset. we present the experimental setup, model description, and comparison with baselines & other methods in section 4. results and error analysis constitutes in section 5. finally, we conclude this paper in section 6. ## related work sentiment analysis was pioneered for text @xcite @xcite but image-based sentiment analysis has got much less attention compared to text-based sentiment analysis. as internet memes is a relatively new research topic, our research work is broadly related to studies on predicting the sentiment from the visual imagery in online reviews @xcite . @xcite introduced sentiment analysis on large scale visual sentiment ontology with sentibank, a system extracting mid-level semantic attributes and from images. to inspect the image posting behavior of social media users @xcite developed the visual emotional latent dirichlet allocation (velda) model to capture the text-image correlation from multiple modalities: the text, visual and emotional perspective of the picture. and interestingly they found results that 66% of users adding an image to their social media posts. recently, anthony @xcite developed a multimodal sentiment analysis approach that combines text and vision to forecast the emotion word tags attached by users to their tumblr posts. @xcite developed a robust image sentiment analysis system by using a cnn on flickr with domain transfer from twitter for binary sentiment classification. benet @xcite worked on a challenge of automatically detect the hate speech in internet memes by using visual information. philipp @xcite developed a multimodal social media system to study how public tweets with images posted by teens who mention gang violence on twitter can be leveraged to automatically discover psycho-social factors and problems. few researchers have investigated to automate the internet meme generation process, while a few others tried to extract its sentiment. to generate popular meme descriptions, @xcite proposed a non-paranormal approach by combining visual and textual features. to produce the meme descriptions @xcite presented an encoder-decoder meme generating system, consisting of a google's pre-trained inception-v3 network to generate an image embedding, followed by lstm model with attention. in this paper, we refer to wang and hua's (2014) method to combine textual and vision information, while scaling up the model using effective dropout regularization. ## dataset we used the corpus provided by task 8 in semeval 2020. this task is named as "memotion analysis". the dataset consists of internet memes and corresponding text descriptions. for task a, each meme is labeled into three sentiment classes: positive, neutral, or negative, and for task b and c, each meme is labeled with several sub-tasks: humorous, sarcastic, offensive, and motivational. ## our approach in this section, we present three different types of deep neural network architectures. we model each task in memotion analysis as a multi-class classification problem where given an internet meme and text description, the model outputs probabilities of it belonging to output classes. the number of output classes will vary for each individual task. the full code of system architecture can be found on github foot_4 . ## conclusion in this paper, we developed a novel multimodal neural network method using deep learning techniques. our model is constructed by concatenating the textual and image feature representations. we discovered that a combination of text and vision modalities give better predictions than single modalities (text or vision) for memotion analysis. till now we handled challenges like the unstructured/elongated words, phrase and word contractions, multiple sentences, noisy data, imbalanced datasets, etc. in future work, we will concentrate on problems like short meme descriptions, free word ordering in sentences, more features to identify expressions in memes, sarcasm in descriptions, etc. we would like to explore more deep neural network architectures that are able to capture humor and sarcasm in internet memes.