NLP _ UIOWA at S em E val-2020 Task 8: You’re Not the Only One Cursed with Knowledge - Multi Branch Model Memotion Analysis (2020)

Abstract: We propose hybrid models (HybridE and HybridW) for meme analysis (SemEval 2020 Task 8), which involves sentiment classification (Subtask A), humor classification (Subtask B), and scale of semantic classes (Subtask C). The hybrid model consists of BLSTM and CNN for text and image processing respectively. HybridE provides equal weight to BLSTM and CNN performance, while HybridW provides weightage based on the performance of BLSTM and CNN on a validation set. The performances (macro F1) of our hybrid model on Subtask A are 0.329 (HybridE), 0.328 (HybridW), on Subtask B are 0.507 (HybridE), 0.512 (HybridW), and on Subtask C are 0.309 (HybridE), 0.311 (HybridW).

## introduction background. with the increasing social media culture, the sharing of internet memes on social media platforms has grown immensely in the recent years. meme is defined as the unit of cultural information that replicates and transmits with reliability and fecundity @xcite . memes are generally an image paired with text, and used to express an array of ideas (e.g. humor, sarcasm). memes can be derived from pop cultures, previous experiences, or even more abstract ideas. memes have become a large part of internet culture, and can preserve viewpoints specific to the community from where it originated. memes can be used to express humor, embarrassment, hate, and even more emotions. the creativity of memes, however, carry a downside. hateful or offensive memes can also be created and can lead to an increase in hate crimes @xcite . as with hateful language, several social media platforms have been working on policies to control such hateful and offensive memes while being careful not to hinder the creativity of users' expressions through memes @xcite @xcite . one of the major steps in controlling the sharing of hateful memes is being able to successfully detect them. detection of offensive content on social media is an ongoing task. current attempts at detecting offensive memes is limited. furthermore, detecting offensive memes is more challenging than detecting offensive text as it involves both visual and language understanding while the latter only requires language understanding. currently, many sites rely on human moderators to identify and remove memes that express emotions that violate the platform's policy. however, with the increasing use of memes across social media platforms, handpicking offensive memes would require larger human resource and can cause problems in scalability. automated systems to identify the emotion of a meme could help in a first line defense/analysis of memes and could help reduce the load on human moderators. we already see this hybrid approach being employed for offensive and hateful text detection on several social media platforms @xcite , so it is only natural to extend this approach to classifying memes as well. in order to address the problem of detecting offensive memes as well as classifying types of memes in general, a group of organizers created a community driven task, semeval 2020 task 8 (memotion analysis). @xcite brings attention of the research community towards automatic meme emotion analysis and allows for the examination of multiple approaches. we approach this problem with a hybrid architecture of convolutional neural network (cnn) for image classification and a bidirectional long short term memory (blstm) neural network for text classification. ## proposed approach our goal is to capture informative features from both images and text to help the system in its classification. to increase the usefulness of both image and text, we first fine tune a cnn on image classification and blstm on text classification separately, then use a validation set to score their respective performances. a cnn was chosen as cnns have shown strong performance in image classification @xcite . likewise, blstms have shown strong performance on text classification, therefore we chose this for our framework foot_0 . we finally combine the cnn and blstm models using a hybrid approach. ## discussion trade off in the performance of blstm and cnn. as seen in ## related work since the multimodal social media content has seen a steady increase in the recent years, deriving the intended meaning from this content by establishing the connection between the image and the text has seen an increase in research. a limited research has been done in extracting meaning from social media images and texts, which includes identification of the humor, offensiveness or sentiment expressed in image, text or meme. humor classification. detection of humor in image and text has been approached by several teams in recent years. @xcite analyze the humor present in abstract scenes at the scene-level and the object level and detect different types of humor depicted in the scenes. @xcite mentions that the meaning and humor in a cartoon is expressed through verbal and visual mode. in order to capture the humor expressed in the cartoon, one has to pay attention to all the verbal and visual details of the cartoon. offense classification. recently, there has been a growing interest in identifying the offensive language of social media data. @xcite presents user-level offensive language detection on social media. this architecture uses features such as the user's writing style, structure, and specific cyberbullying for detecting offensiveness in the text. @xcite proposed a germeval task for classifying offensive language as offensive, or other, and then further classify the offensive tagged language. more recently, @xcite ran a shared task, offenseval, on detecting different classes of offensive text. sentiment classification. sentiment detection in the image or text has also seen a greater focus on research. @xcite mention that the accurate sentiment detection from internet images requires connection between visual and textual feature. they presented the unsupervised sentiment analysis (usea) framework to perform sentiment analysis on social media images in an unsupervised approach using both features mentioned earlier. @xcite present a method built upon web mining to automatically construct a visual detector library to detect adjective noun pair in an image, which they used to identify the sentiment from visual content. sarcasm classification. though sarcasm is not always easy to identify online, researchers have attempted this with various approaches. @xcite present a survey on various methods for automatic sarcasm detection. they link to many sarcasm papers which include the sarcasm datasets used (e.g. @xcite ) as well as sarcasm detection approachs leveraged (e.g. @xcite ). ## conclusion we analyzed texts and images from memes using blstm and cnn, respectively. we then propose two hybrid approaches hybride (equal weightage to prediction probability from blstm and cnn) and hybridw (weighted average based on performance of blstm and cnn) to identify humor, offensiveness, and sentiment expressed in memes. hybride performs better overall than the individual systems,