KAFK at S em E val-2020 Task 8: Extracting Features from Pre-trained Neural Networks to Classify I nternet Memes (2020)

Abstract: This paper presents two approaches for the internet meme classification challenge of SemEval-2020 Task 8 by Team KAFK (cosec). The first approach uses both text and image features, while the second approach uses only the images. Error analysis of the two approaches shows that using only the images is more robust to the noise in the text on the memes. We utilize pre-trained DistilBERT and EfficientNet to extract features from the text and image of the memes respectively. Our classification systems obtained macro f1 score of 0.3286 for Task A and 0.5005 for Task B.

## task description the memotion analysis @xcite challenge is comprised of the three tasks listed below. the metric of evaluation for task a is macro f1. for task b and c, the evaluation metric is the average of the macro f1 for each of the sub-tasks. we participated in task a and task b. ## data the dataset is compiled by @xcite . it contained 6992 memes labelled for each of the tasks mentioned in section 2. typically, memes are composed by overlaying text on images. hence, the dataset aptly consists of images and their corresponding (ocr extracted) overlay-ed texts. the text is in the english language. the amount of samples for task a and b are given in ## results we report the performance of the best models from our experiments in the official rank and macro f1 score of our models on the test set are given in ## error analysis although, system 1 was better at classifying the memes that had longer and more coherent sentences, it failed to classify memes that had short and incoherent sentences. due to higher number of positive samples in the dataset, this system developed a tendency to classify the memes as positive in task a. system 1 performed poorly on memes that had multiple panels or when two or more memes were stitched together. we also found that many of the samples contained texts that were unrelated to the context of the meme such as web addresses which might have thrown off the text-based classifier. such unrelated text were often just for the promotion of the sites or users that host or create these memes. for example, "9gag fat cat memes funny -9gag.com", etc. and being random in nature, it is difficult to remove such unnecessary parts/noise from the texts. but these samples were somewhat more correctly classified by system 2 which uses only the images. also, since system 2 does not use text, it failed to classify the samples where most of the context was in the text. ## conclusion during the course of our experiments, we find that there are multiple challenges in the classification of internet memes. memes don't adhere to any standards and they only continue to evolve. we tested out two approaches for classifying memes, using both text and image together and using only the images. error analysis showed that the image-only approach is somewhat robust to noise in the text but fails to capture the context conveyed by the text. we suspect this task will get more difficult when we take into account memes that use video and sound. in future work, it will be interesting to explore pre-trained simclr models @xcite . these models are trained to learn generic representations from images in a self-supervised manner, which can be used to improve classification tasks.