Codewithzichao@ D ravidian L ang T ech- EACL 2021: Exploring Multimodal Transformers for Meme Classification in T amil Language (2021)

Abstract: This paper describes our submission to shared task on Meme Classification for Tamil Language. To address this task, we explore a multimodal transformer for meme classification in Tamil language. According to the characteristics of the image and text, we use different pretrained models to encode the image and text so as to get better representations of the image and text respectively. Besides, we design a multimodal attention layer to make the text and corresponding image interact fully with each other based on cross attention. Our model achieved 0.55 weighted average F1 score and ranked first in this task.

## introduction in recent years, with the prosperity of social media platforms, memes have gradually become a part of online communication. therefore, it is essential to detect whether memes are offensive to individuals or organizations to ensure the diversity and sustainability of content on the internet. it it a challenging task to classify whether memes are troll or not. in addition, there has been a lot of work currently focused on english @xcite @xcite , but little work has been done for tamil language. shared task on meme classification for tamil language fills this gap. the goal of this shared task is to detect whether memes which are collected from social media platforms are troll or not. each meme has been annotated with troll or not troll class. furthermore, a transcription of captions in latin script for both tamil is embedded in each image. this is a multimodal classification task that given the image and text pair, systems have to classify this pair into troll or not troll class. in this paper, we explore a multimodal transformer for meme classification on tamil language. according to the characteristics of the image and ## data the data we used is provided by the organizers of shared task on meme classification in tamil language @xcite . there are 2300 samples in the training data. the specific statistics of the data are shown in ## proposed model in this section, we will present our model for meme classification on tamil language. our model is mainly divided into three layers: encoding layer, multimodal attention layer and prediction layer. encoding layer is used to obtain word representations and image representations. multimodal attention layer is used to make the text and corresponding image interact fully with each other. prediction layer is used to get the probabilities of all classes. overall model architecture is shown in figure 1 . ## conclusion in this paper, we present a multimodal transformer for meme classification on tamil language. using resnet and xlm-roberta for the image and text, we obtain better representations of the image and text. besides, we design a multimodal attention layer to make the text and corresponding image interact fully with each other. finally, our model took first place in this task which demonstrates the effectiveness of our model. in future research, we will explore ways to better filter irrelevant information in the image and text to obtain better performance.