Findings of the Shared Task on Troll Meme Classification in T amil (2021)

Abstract: The internet has facilitated its user-base with a platform to communicate and express their views without any censorship. On the other hand, this freedom of expression or free speech can be abused by its user or a troll to demean an individual or a group. Demeaning people based on their gender, sexual orientation, religious believes or any other characteristics –trolling– could cause great distress in the online community. Hence, the content posted by a troll needs to be identified and dealt with before causing any more damage. Amongst all the forms of troll content, memes are most prevalent due to their popularity and ability to propagate across cultures. A troll uses a meme to demean, attack or offend its targetted audience. In this shared task, we provide a resource (TamilMemes) that could be used to train a system capable of identifying a troll meme in the Tamil language. In our TamilMemes dataset, each meme has been categorized into either a “troll” or a “not_troll” class. Along with the meme images, we also provided the Latin transcripted text from memes. We received 10 system submissions from the participants which were evaluated using the weighted average F1-score. The system with the weighted average F1-score of 0.55 secured the first rank.

## introduction we have seen a rise in the usage of memes on the internet. memes could come in many forms and languages, but we emphasize on the image with text (iwt) memes @xcite in the tamil language. iwt memes are inherently multimodal as their meaning is not understood when just the image or the text considered in isolation; hence, both image and text should be considered. these memes can propagate and mutate through cultures like the selfish gene @xcite . propagation and mutation of memes ensure widespread on the internet; hence they could be weaponized by trolls @xcite . trolls are individuals or users on the internet who tend to attack or offend other individuals or groups in a demeaning manner @xcite . a meme used by troll -troll meme-could come in any languages, and since languages are representative of cultures, a meme in one language could represent a culture. in this shared task, we provide the tamilmemes dataset @xcite ) that consists of 2,969 memes to the participants. additionally, we also provided the text (caption) associated with each meme. the text has been transcripted in latin by a native tamil speaker, which was later evaluated and corrected by the expert. tamil (iso 639-3: tam) is an official language in tamil nadu, puducherry, sri lanka, singapore and a recognised minority language in malaysia and south africa with a 75 million speaker base @xcite @xcite . tamil inscription in pottery from kodumanal and porunthal foot_0 foot_1 was the oldest inscription in india dating to 580 bce then asoka inscription in prakrit, greek and aramaic dating to 260 bce @xcite . in sanskrit, the oldest known inscriptions are from the 1st century ce, such as dhana's ayodhya inscription and ghosundi-hathibada. the first book printed in india in an indian language was tampiran vanakkam in tamil on 1578 ce, a 16-page translation of the portuguese "doctrina christam" @xcite . in the modern tamil script, there are 12 vowels, 18 consonants and one special character, the āytam (chakravarthi, 2020b). the vowels and consonants combine in order to form 216 compound characters, giving a total of 247 characters. however, tamil is often transcribed in latin script by a multilingual speaker on social media platforms @xcite . this behaviour could be attributed to the ease or comfort in scripting tamil in latin script @xcite . also, these multilingual speakers tend to switch to another language such as english. this phenomenon, also known as code-mixing or codeswitching is commonly observed amongst multilingual speakers @xcite . a great deal of work and resources (chakravarthi et al., 2020a,c) has been created for dravidian languages but the multimodal aspect of code-mixed content in the form of meme remains unexplored. in recent years, we have seen a rise in awareness of tackling multimodal offensive @xcite or hate @xcite content in memes in the english language, but other under-resourced languages remain unexplored. our "troll meme classification in tamil" task aims to promote research in multimodal troll meme classification in under-resourced tamil language. moreover, this task provides a unique opportunity to study the effect of code-switching or code-mixing in the english transcript of tamil @xcite in association with the image from the meme. ## task description the goal of the "troll meme classification in tamil" shared task was to classify if a given meme is a "troll" or "not-troll" based on the image and text associated with the meme in the tamil language. the text from the meme is written in either the tamil grammar and english lexicon or english grammar and tamil lexicon. however, for consistency, we transcripted the text in latin. troll meme is a meme, which consists of offensive text and non-offensive images, offensive images with non-offensive text, sarcastically offensive text with non-offensive images, or sarcastic images with offensive text to provoke, distract and has digressive or off-topic content with intend to demean or offend particular people, group or race, otherwise, a not-troll meme @xcite . figure 1 shows examples of a troll and nottroll meme from the tamilmeme dataset. example 1 is a troll meme targeted towards the potato chip brand called "lays". in this example, an image is harmless with just a picture of the potato chips previously, we developed this tamilmemes dataset and treated the task of identifying a troll meme as an image classification problem. since the text associated with the meme acts as a context of the image, we enhanced our tamilmemes dataset by providing the text as a separate modality for the shared task. we expected our participant to approach the task in a multimodal way. rank team ## evaluation the weighted average f1-score 3 is calculated by averaging the support-weighted mean f1 score perclass. ## methodology in system submissions, we saw a variety of methodologies used by the participants which were as simple as logistic regression @xcite and as complex as bert @xcite . by keeping multimodality in mind, most of the participants implemented a method that could leverage both text and image. mostly, the image part has been processed using convolutional neural network (cnn) such as resnet152 @xcite or the custom residual networks while the text has been processed rnn (lstm @xcite ), transformer (bert, roberta @xcite ). however, in one exceptional case, one participant has used a transformer for processing both modalities. their system submission for tamil troll meme classification achieved a 0.54 weighted average f1 score. ## results and discussion we have received 10 submissions for the shared task. in this shared task, we provided 2,967 samples which might be less for the data-hungry models such as bert and it's predecessor such as roberta, xlm-roberta. moreover, due to the high dimensionality presented by the images, the traditional machine learning models such as logistic regression, naive bayes are not able to generalize. the poor performance of the traditional machine learning model shows that they are rather too simple to learn useful features from the high dimensional data presented in the form of image and text. furthermore, the tamil text (could be with or without code-mixed) from the meme is transcripted in the latin script leaves little room for using pre-trained models (e.g. multilingual bert) which are primarily trained in english or zero-shot transferred to other languages written in their native script. hence, such multilingual models need to be finetuned on the more code-mixed, transliterated data. in the case of images, the visual features have been captured using a cnn pre-trained on the imagenet weights. in the previous study, we observed that the visual features captured by such cnns tend to be less useful for identifying a troll tamil meme. the same effect has been observed in the system submitted to the shared task. overall, we need a robust approach that takes into account all the aspects of this multimodal classification problem -lack of data, high dimensionality, low resource language processing and code-mixing. ## conclusion in the "troll meme classification in tamil" shared task, we presented a unique multimodal classification problem by providing a newly improved tamilmeme dataset which now has tamil text (in the form of transcripted latin text) from memes. this task was not only a multimodal classification problem but also posed challenges such as natural language processing of low-resourced language, transcripted code-mixed text. the submissions received from the participants showed multiple ways to approach the problem and we hope that their contribution along with our enriched dataset will kindle the research in this less explored area.