UVCE - IIITT @ D ravidian L ang T ech- EACL 2021: T amil Troll Meme Classification: You need to Pay more Attention (2021)

Abstract: Tamil is a Dravidian language that is commonly used and spoken in the southern part of Asia. During the 21st century and in the era of social media, memes have been a fun moment during the day to day life of people. Here, we try to analyze the true meaning of Tamil memes by classifying them as troll or non-troll. We present an ingenious model consisting of transformer-transformer architecture that tries to attain state of the art by using attention as its main component. The dataset consists of troll and non-troll images with their captions as texts. The task is a binary classification task. The objective of the model was to pay more and more attention to the extracted features and to ignore the noise in both images and text.

## introduction over the past decade, memes have become a ubiquitous phenomenon over the internet. memes can come in several formats such as images, video, etc. memes can take a combined form of both text and images too. due to its vast popularity, different people perceive memes distinctively. recent studies have prompted the usage of memes as a mode of communication across social media platforms. the presence of text in images makes it harder to decode the sentiment or any other characteristic @xcite . regardless of the type of the meme, they may be changed, recreated over social media networks, and tend to be used in contexts involving sensitive topics such as politics, casteism, etc, to add a sarcastic perspective @xcite . due to its multimodality, conscientious analysis of memes can shed light on the societal factors, their implications on culture, and the values promoted by them @xcite . in addition to that, analyzing the intended emotion of a meme could help us acknowledge fake news, offensive content that is being propagated using the internet memes as a medium, thus helping in eradicating the spread of misinformation and hatred to the large user base in social media @xcite . it is plausible that memes might become an integral part of most of the people, as it is used to understand racial and gender discourse on social media platforms such as reddit @xcite . one of the approaches to overcome this is manually monitoring and moderating user-generated content. but due to the amount of data being generated on the internet every day, it would be ideal to develop automated systems to moderate them @xcite @xcite @xcite . consider countries with huge populations such as india, several memes are directed towards targeted communities. to address these issues of identifying if a given meme is trolling a person's sentiments, a dataset for memes that were suspected to troll a particular community. we participate in the shared task on meme classification based on the troll classification of tamil memes @xcite . tamil (iso 639-3: tam) language is spoken in south asia @xcite . the earliest inscription in india dated from 580 bce was the tamil inscription in pottery and then the asoka inscription in prakrit, greek, and aramaic dating from 260 bce. the earliest known inscriptions in sanskrit are from the inscriptions of the 1st century bce. tamil is the official language of tamil nadu, india, as well as of @xcite . the task primarily consists of identifying whether a meme is a troll or a non-troll @xcite . we use the images and captions that are provided to achieve the most efficient model to classify the memes. we use a combination of vision transform (vit) @xcite and mbert @xcite over other pretrained models used for image classification as described in @xcite . ## related work internet memes have been a subject of interest for both computer vision and natural language processing researchers. the type of memes that are being used illustrates the context of discussions on social media platforms. people are using memes to express themselves, and in the making, showcase their stance on a certain social issue, be it in acknowledgment or rejection of the issue @xcite @xcite . there exist several reasons that suggest the spread of memes. some of the reasons include novelty, simplicity, coherence. it also includes an emotional attachment, its ability to have different meanings, depending on how a person perceives it @xcite @xcite . hu and flaxman developed a multimodal sentiment analysis by developing a deep neural network that combines both visual analysis and text analysis to predict the emotional state of the user by using tumblr posts. ## data we use troll classification dataset of tamil memes @xcite . it consists of 2,699 memes, of which most of the images have text embedded within them. we are also provided with captions for all images. the distribution is shown is ## system description multimodal deep learning is a robust and efficient way of addressing the main goals of artificial intelligence by integrating and combining multiple communicative modalities to obtain crucial results which usually improves the outcome of the single models trained. as deep learning models tend to extract features on their own, the objective can easily be achieved with the help of neural networks. given the images of tamil memes, along with the embedded text on the images, scrutiny of images and texts independently and then picking out relevant information for further process plays a climacteric role in our system. at the end of the training, the model has to output a single value stating the given meme is troll or non-troll. the specialty of our model was to neither use the convolutional neural networks (cnn) nor recurrent neural networks (rnn). as the title of the paper points out, the model tries to gain more attention towards the salient portions of text and images. the proposed solution makes an effort to convey the importance of attention gain and its relation to the performance of the model. the model is put forward to compute the classification is vision transformer @xcite for images and bidirectional encoder representations from transformers (bert) @xcite for captions of memes. this corresponds to a transformer-transformer architecture as shown in fig 1 . ## experiments all suitable models were implemented using py-torch version 1.5.0 in a google colaboratory environment. the early stages of this model include preprocessing of images. the dataset had pictures with various resolutions and had to be made equal. the images were resized to 256 x 256 pixels. most of the images had texts on the top and bottom of the images. texts in the images were considered as noise for classification, which resulted in performing a center crop for all images. the border of the portions was removed and images of size 224 x 224 were produced. finally, the images were ready as the input to the transformer by normalizing the rgb channels with mean 0.485, 0.456, 0.406, and standard deviation 0.229, 0.224, 0.225 respectively. no augmentations were made to preserve the meaning of the images. the transformer was originally trained on the imagenet dataset and had achieved remarkable results. the trained weights are transferred to this downstream task. the base version of vit is fine-tuned which had default hyperparameters of 16 patches, an embed dimension of 768, 12 layers, 12 attention heads, and a dropout rate of 0.1. the head of the vision transformer, which outputs 1000 classes, is now replaced by a linear layer of 128 neurons. the texts were also preprocessed by removing stopwords, special characters, and punctuation. texts need to be tokenized before feeding into the bert configuration. after inserting it into the transformer, the resulting pooled output from the multilingual bert model is also passed through a linear layer of 128 neurons. the two layers obtained from the transformers are merged together to form a single layer with 256 neurons. this is passed through the relu activation function and a dropout to obtain one final neuron which determines the class as troll or nontroll. a learning rate of 2e -5 was used with a batch size of 16. the maximum length of the captions was truncated to 128 as memes usually do not contain very long sentences. the training was done for 4 epochs and with a linear schedule with warmup. to our surprise, the model learned very rapidly and achieved well progress on the validation set which mimicked the train set. it was also observed that merging the outputs of two different domain models did not harm the training, moreover, it helped in getting better results. ## results we achieve an overall f1-score of 0.96 when we use images for classification using vit as shown in 2. it is to be noted that using mbert to clas-sify memes solely based on the captions achieves 0.93 as f1-score as shown in ## conclusions the proposed solution performs at greater heights on the validation and set in the training phase. the validation set mimics the train set as the memes are split looking at the distribution of the classes. the dataset is very small and augmenting it will not help for the optimal results. the algorithm overfits the train set undoubtedly. the reason behind the poor performance is due to the change in the distribution. the memes in the test set had multiple images which were difficult for the vit to capture features. the model scored a f1 score of 0.46 on the test set and 1.0 on the validation set. vast difference can be observed due to high bias. here, in this paper, we have tried to come up with this innovation of transformer-transformer architecture which can achieve extreme results. in the future, we will be performing a wonderful task of having more transformers in parallel computation and syncing them makes an immense difference in this era of deep learning.