NIT -Agartala- NLP -Team at S em E val-2020 Task 8: Building Multimodal Classifiers to Tackle I nternet Humor (2020)

Abstract: The paper describes the systems submitted to SemEval-2020 Task 8: Memotion by the ‘NIT-Agartala-NLP-Team’. A dataset of 8879 memes was made available by the task organizers to train and test our models. Our systems include a Logistic Regression baseline, a BiLSTM +Attention-based learner and a transfer learning approach with BERT. For the three sub-tasks A, B and C, we attained ranks 24/33, 11/29 and 15/26, respectively. We highlight our difficulties in harnessing image information as well as some techniques and handcrafted features we employ to overcome these issues. We also discuss various modelling issues and theorize possible solutions and reasons as to why these problems persist.

## introduction over the years, the internet and social media have become an indispensable part of our lives. today, an average netizen spends over 45 minutes on some form of social media everyday 1 . therefore, social media has become a goldmine for data to model and study human opinions and behaviour. social media analysis conventionally deals with data in the form of text, audio or video -but often only one prominent modality is studied in isolation. this lack of multimodality in research works has led to entire modes of communication being disregarded. one such form of internet communication that we have yet to tap into is memes. while a meme was initially defined as "an idea, behaviour, or style that spreads from person to person within a culture-often to convey a particular phenomenon, theme, or meaning 2 ," it has transformed into the umbrella term for a suite of referential humor that plagues the internet. memes can more aptly be defined as "a form of referential humor that incites humor by leveraging images, text and sometimes audio." the image is more often than not coupled with a real-life event or media. the images are then re-purposed to incite humor through the subversion of expectations and via reference. through the years, memes have been leveraged differently by various communities -apart from innocent, humor inducing purposes. corporations and companies are increasingly interested in harnessing memes to sway their younger customers. more adversely, certain memes such as pepe the frog 3 were adopted by alt-right groups and used as a calling card of sorts. memes are also used as performative acts, which involve a conscious decision to support or oppose a movement (as seen in the case of the recent hong kong protests 4 ). the abundance of memes on social media platforms such as facebook, instagram, and twitter @xcite further suggests that -these digital constructs have become a flagship of internet culture, so and so that, to understand memes would mean to understand the views of a community. the semeval task 8: memotion @xcite shared task is one such grassroots endeavour to understand memes better and incorporate multimodality in social media analysis. apart from the rich information that could be derived from understanding social media, solving the various challenges of the task also pose unique research value -such as the training of models that could understand reference derived from background knowledge and complex human expressions such as humor and sarcasm. in this paper, we share the insights we gain from our time working with the task and dataset. in section 2, we discuss some previous work and related approaches to tackling the problem. then, in section 3, we introduce the task and the dataset. subsequently, in section 4, we outline the features employed and the model architectures. section 5 reports our results and model performance. in section 6 and 7, we analyze our results and discuss the possible source and solutions to our errors and issues. we summarize our key results and conjectures in section 8. ## related work research on memes, in particular, are few and far in between. some works have experimented with memes as sentiment predictors: in their work on facebook sentiment analysis, @xcite reported a positive correlation between the category of the meme used and the affection of the discussion from its texts. french (2017) go on to confirm that memes were far more successful in conveying the sentiment of the debate over textual data. other works attempt to automate the meme generation process @xcite but fail at replicating the humor expressed by particular memes. while peirson and tolunay (2018) attempts to re-purpose image captioning algorithms to capture the humor, @xcite attempt to use macros within news headlines to do the same. however, both works fail to generate coherent humor that can persuade annotators consistently. another similar work by wang and wen (2015) apply multimodal techniques to generate captions for popular meme formats. finally, the propagation and effect of memes on social media are also studied in works such as the one by @xcite which provides an assessment of the popularity and use of certain memes, in the context of each community. in another work, @xcite aim to detect and analyze meme usage in social media streams, particularly by using an unsupervised clustering framework. work in the space also make use of meta-information @xcite regarding the meme to provide context to their frameworks. apart from memes, other work on multimodal social media analysis and humor analysis could provide valuable insights. @xcite show how multimodal information can be used to enhance the analysis of unstructured data on social media. there are numerous works that attempt to predict the sentiment of images @xcite . however, these works focus on a single modality -images. works on multimodal sentiment analysis like the one by @xcite use both image and text embeddings in their models to show slight improvements over the text-only models. a comprehensive overview of multimodal sentiment analysis can be found in work, such as that @xcite and @xcite . in the realm of humor analysis -work such as the recent paper by @xcite have shown that deep learning models can outperform humans in classifying humor mainly due to a disparity in the sense of humor of the annotators and the testers. in an interesting work by @xcite an attempt is made to capture wit by using synonyms of words in image descriptions to incite puns through the subversion of expectations. while the work reports impressive results and beats out humans in a controlled vocabulary setting, humans quickly regain dominance when the vocabulary is unconstrained. ## data the task organizers have made available a dataset @xcite of 8879 annotated memes scrapped from various sources across the internet. each meme was annotated by two annotators to ensure annotation quality. the text was extracted from the image using the google ocr system and manually corrected by crowdsourced workers -to ensure that model accuracy doesn't depend on the quality of the ocr techniques used. we briefly describe each subtask of the memotion shared task below: the dataset shows a significant imbalance, particularly in the representation of the labels negative, hilarious, very twisted and hateful offensive of the sentiment, humor, sarcasm and offensive categories, respectively. we represent the distribution of labels in ## preprocessing and system overview before we outline our features and models, we digress to briefly explain the steps we undertook to reduce noise in the text. we perform basic lower casing and the removal of unnecessary punctuation as the initial step. the second step was to remove specific noise inducing aspects such as the removal of urls and user mentions (using regular expressions). we did not perform any global image preprocessing steps; however, we did perform basic preprocessing steps such as resizing and grayscale conversion for specific feature extraction techniques. ## experimental setup and results the dataset was provided to task participants as a pre-annotated training dataset (containing 7001 samples) and an un-annotated test dataset (containing 1878 samples). we first perform a model and ablation analysis using 10-fold cross validation methods on the training dataset and follow up with scores obtained by our systems on the test dataset, as provided by the task organizers. all results are represented using accuracy and macro f 1 metrics which were used for ranking the systems by the task organizers. for the training dataset, initially, a validation dataset is also maintained to diagnose variance and bias issues that arise in the training phase and to aid hyperparameter tuning. our overall train, validation, test split ratio is 80:10:10. however, for our final results, we conflate the validation and training set and represent 10-fold cross validation results. we represent our results for subtask a and the categories within subtask c (as subtask b and c only vary by semantic levels) in our model study revealed that logistic regression and the bert model exhibit the best results, trading places based on what metric was considered. the results of the attention based learner were underwhelming and were not competitive with the other models. our ablation analysis reveals that in general balancing and augmentation techniques provide the biggest performance improvements (on the basis of macro f 1 ). we also note that both these techniques were implemented to address the data scarcity and imbalance issue. image features too, provide slight improvements when added (in all combinations). however, when these techniques are considered in combination, the addition of augmentation over text + balancing + image features, lead to a drop in performance for the humor category and also in the sarcasm task, where the text + image features model outperforms the text + augmentation + image features model. in an additional test, we also observed that for the attention based learner, image information provided little to no improvement on the performance. on the test dataset, again bert and the logistic regression model obtain the best results. we also observed that the attention based learner exhibits lower performance on addition of image features, on all sub tasks. ## error analysis due to the under-representation of specific labels, we saw our models were unable to classify samples into these classes effectively. this was especially apparent for the highly under-represented hateful offensive (of the offensive category) and very twisted (of the sarcasm category) where very few samples could be correctly classified. while we have reservations regarding the subjective nature of humor and offense, we generally found offensive memes wrongly classified as non-offensive and funny memes wrongly classified as not funny when the context mostly derived from the image. this points to our inability to capture image information effectively. this also indicates the absence of any background knowledge, which is imperative when understanding references that most memes tend to invoke. on analyzing the errors and ground truth of the motivation category, we find many of the annotations puzzling. we are, therefore, more curious as to what the annotation guidelines for this category are and refrain from making any conjectures regarding the classification errors. another issue we would like to explore is the underwhelming results of the bilstm + attention model. the main issues faced during the training phase are the small overall dataset size and under-representation of specific labels. this led to the inability to train larger and deeper models without facing high variance issues. another issue that arose due to the small dataset size is the inability to train an image feature extractor from scratch (to alleviate data dissimilarity between the dataset and pretrained models) and better harness image information. ## discussion in our time with the task of meme emotion analysis, we have come to believe that there are many inherent challenges in modelling the task apart from the machine learning algorithms being leveraged. we want to bring attention to the fact that memes are ever-changing -new memes enter and old memes leave the meme ecosystem very frequently and are quite often informed by current affairs. this could imply that training a model on historical meme samples could provide little insight into the meme ecosystem of today. the trends learnt by training with historical samples might not translate well to the trends of today. in the data collection and annotation phase, we make the following observations: meme heterogeneity: memes come in different forms and styles -or so-called templates. these templates can either represent a certain punchline or message and may not share the same referential image. for example, the set of memes {"the scroll of truth" foot_5 , "nancy pelosi ripping paper" foot_6 ,"most people rejected his message" foot_7 } represent the same idea of 'ignoring a fact or supposed truth' but use different referential images to enhance the humor. memes belonging to the same template can be assumed to convey a similar message. there exists a countless number of such templates that are used online. therefore, we think datasets should not attempt to draw memes from all templates but certain ubiquitous ones -as in the former case, the dataset does not contain enough samples to adequately represent the meme template's idea. as a product of this, models face the insurmountable task of learning trends from a large number of templates, each with a minimal set of samples. it may be apt to reverse this idea and create datasets with few selected templates but a large number of samples corresponding to each template. ## conclusion our work on this dataset highlights the difficulty in effectively harnessing image information to understand reference and media that are part and parcel of a meme. consistent with previous work, we are only able to obtain small performance improvements by employing image-based features. we also highlight modelling issues that arise due to the heterogeneous nature of data (a small number of samples, a large number of templates) and the high variance issues. we also bring to question -annotation and data collection practices that might not immediately translate to the task of meme analysis. a general suggestion would be to use a large number of annotators or to harness popularity metrics such as likes and upvotes used on websites like reddit and instagram. popularity metrics could provide a more generalized view of the humor or other characteristics of the meme. we think that memes can be imperative to understanding user sentiment in the present internet ecosystem and beyond. we look forward to the new interest that enters this space due to work done here and in the memotion shared task.