Findings of the Shared Task on Multimodal Sentiment Analysis and Troll Meme Classification in D ravidian Languages (2022)

Abstract: This paper presents the findings of the shared task on Multimodal Sentiment Analysis and Troll meme classification in Dravidian languages held at ACL 2022. Multimodal sentiment analysis deals with the identification of sentiment from video. In addition to video data, the task requires the analysis of corresponding text and audio features for the classification of movie reviews into five classes. We created a dataset for this task in Malayalam and Tamil. The Troll meme classification task aims to classify multimodal Troll memes into two categories. This task assumes the analysis of both text and image features for making better predictions. The performance of the participating teams was analysed using the F1-score. Only one team submitted their results in the Multimodal Sentiment Analysis task, whereas we received six submissions in the Troll meme classification task. The only team that participated in the Multimodal Sentiment Analysis shared task obtained an F1-score of 0.24. In the Troll meme classification task, the winning team achieved an F1-score of 0.596.

## introduction people use different modes of content, including video, audio and text, to express their opinion or attitude or interact with another person. these types of data are too complex to process because of the ambiguity at various levels. moreover, such types data are more user-centric and contextual @xcite . the complexity of the computational processing of social media is more for multimodal data, which includes video, audio and text modalities. the machine learning or deep learning model should utilize/extract the features identified from all modalities for better decision making @xcite @xcite @xcite @xcite . this shared task on multimedia social media analysis in dravidian languages includes two subtasks -multimodal sentiment analysis and troll meme classification. sentiment analysis is a natural language processing (nlp) task to identify the underlying sentiment or opinion about movies, products, government policies etc., expressed by people through various social media platforms @xcite , @xcite . nowadays, social media users use multiple modalities such as images containing text and videos to express their opinions. it has become common to share review videos about movies and products on youtube, and facebook @xcite , @xcite . therefore, analysis of multiple modalities has become significant in identifying sentiments from video data. video data contain modalities such as video frames, speech signals, and text (transcripts). training a machine learning model requires considering the features of the above three modalities to train a machine learning model. the shared task on multimodal sentiment analysis aims at inviting researchers for developing machine learning models to draw out the sentiments from movie review videos in malayalam and tamil. malayalam and tamil are members of the dravidian language family @xcite @xcite @xcite . in addition to that, both languages are morphologically rich and agglutinative @xcite , @xcite . this makes these languages complex for computational processing @xcite @xcite @xcite . moreover, the complexity increases when we deal with speech and text for learning feature representation from video data. we created the dataset @xcite ) by downloading movie review videos that have been uploaded on youtube. the malayalam dataset consisted of 70 videos, and the tamil dataset contained 64 videos in it. videos are classified into five categories: highly positive, positive, neutral, negative and highly negative by considering the facial expressions of the reviewers and the words used to give a review for the movies. memes have become prevalent in social media in recent times. people use memes, which can come in image and video modalities, to express their opinions or attitude on various issues @xcite . some memes are created only for entertainment purposes. however, certain users use memes to inflame individuals or organizations. the computational processing of the meme is challenging because the text that appears in an image or video is difficult to extract @xcite . in addition to that, the text may appear in different fonts and sizes, increasing the complexity. the objective of the troll meme classification shared task is to classify a troll meme into either troll or nottroll categories. we collected troll memes from social media platforms, namely facebook, instagram, whatsapp, and pinterest @xcite . this paper presents an overview of the shared task on multimodal sentiment analysis and troll meme classification in dravidian languages. we considered data collected from both malayalam and tamil for these two tasks. this work also discusses the results of the participating teams. we shared the training and validation data with labels and test data without labels with the participants. the participants submitted the predicted labels for the test data by building machine learning models for the task. in total, 56 teams registered for the multimodal sentiment analysis competition organized by codalab foot_0 . however, only one team submitted their results for tamil test data, whereas six teams participated in the troll meme classification task. the paper is organized into four sections, including the introduction. section 2 describes the multimodal sentiment analysis shared task and the dataset used in detail. it is followed by section 3, which summarizes the troll meme classification task. in section 4, we discuss the systems and methodologies submitted by participants for both shared tasks and analysis of the results of submitted models. a summary of the tasks and future work are presented in section 5. ## multimodal sentiment analysis task description artificial intelligence models that can recognize emotion and opinions are helpful for a variety of industries to understand user requirements, preferences and reviews. from virtual assistants to content moderation, sentiment analysis has many applications. a great deal of multimodal content has been published in local languages on social media about products making the task of sentiment analysis challenging @xcite . the reach of visual information and the traction of smartphones have facilitated people to use videos for sharing their opinions. hence, our task aims to identify highly positive, positive, neutral, negative, and highly negative videos by taking the video, speech and text modalities. ## troll meme classification task description this shared task aims to devise methodologies and models for detecting troll memes in tamil using textual and visual features. a troll meme consists of offensive text and non-offensive images and vice versa, intending to degrade, provoke, or offend a person or a group. a dataset with two classes, namely troll and not troll, has been provided for this shared task. the dataset has images associated with captions and comprises memes and transcribed text in latin, which are annotated and transcribed. the participants used images, captions, or both to do the classification. a sample troll and non-troll meme has been shown in figure . the task organisers have enhanced the dataset by giving the text as a separate modality for the shared task because the text linked with the meme acts as a context for the image. both text and imagebased classification approach, that is, multimodal classification, was expected from the participants. ## conclusion we presented two multimodal shared tasks -multimodal sentiment analysis and troll meme classification in dravidian languages. multimodal sentiment analysis in dravidian languages is the first shared task in this area. in addition to the multimodality, the code-mixing nature of the language posed challenges in these two tasks. we created datasets for both tasks to aid the research in the under-researched area. the machine models used by the participants revealed different ways of solving the problems mentioned above. we received one submission in the multimodal sentiment analysis task and six submissions in troll meme classification.