UPB at S em E val-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis (2020)

Abstract: Users from the online environment can create different ways of expressing their thoughts, opinions, or conception of amusement. Internet memes were created specifically for these situations. Their main purpose is to transmit ideas by using combinations of images and texts such that they will create a certain state for the receptor, depending on the message the meme has to send. These posts can be related to various situations or events, thus adding a funny side to any circumstance our world is situated in. In this paper, we describe the system developed by our team for SemEval-2020 Task 8: Memotion Analysis. More specifically, we introduce a novel system to analyze these posts, a multimodal multi-task learning architecture that combines ALBERT for text encoding with VGG-16 for image representation. In this manner, we show that the information behind them can be properly revealed. Our approach achieves good performance on each of the three subtasks of the current competition, ranking 11th for Subtask A (0.3453 macro F1-score), 1st for Subtask B (0.5183 macro F1-score), and 3rd for Subtask C (0.3171 macro F1-score) while exceeding the official baseline results by high margins.

## introduction the internet represents the biggest source of knowledge humankind currently possesses. people from all corners of the world can express their opinions, thoughts, and share their insights regarding certain ideas. during the past decade, a new and never seen before way of sharing beliefs arose, memes. for example, by humorously combining text and images, their authors can emphasize a series of aspects such that they will amuse, in various degrees and ways, the receptors. ## proposed approaches we propose three multi-task learning architectures to solve all the subtasks of the memotion analysis competition (i.e. sentiment classification, humor classification, and scales of semantic classes). because the entries are bimodal, the resulting system can be divided into two main components based on their input (i.e. one for image-only and the other for text-only). both parts will act as a feature extractor, the resulting encodings being fused together to obtain the text-image representation. based on the fused features, our system will discriminate between the output classes of each subtask. next, we offer details regarding the three systems. ## conclusions and future works this paper presented our approaches regarding the memotion analysis shared task, organized by semeval-2020. we proposed several architectures that intends to solve the memotion analysis issue by using recent breakthroughs in the computer vision, as well as the natural language processing field: vgg-16 alongside albert. by creating multi-task learning solutions, joint textual and visual modeling network, and separate channels, as well, we were able to achieve good scores for the previously mentioned subtasks and provide a good insight on the way this relatively new challenge can be approached. we highlighted that factors such as image noise, incorrect typing, or overlapping text are key parts that stand against a proper analysis. also, various combinations of texts and images can cause serious problems in the analysis process conducted by deep learning approaches. in the future work, we will investigate the effect of using improved visual models, such as the enhancement for vgg-16, vgg-19 @xcite . furthermore, considering that albert is a lite bert, we also intend to explore text processing models containing a larger number of parameters, such as bert-base, or even bert-large.