Findings of the Shared Task on Multimodal Abusive Language Detection and Sentiment Analysis in T amil and M alayalam (2023)

Abstract: This paper summarizes the shared task on multimodal abusive language detection and sentiment analysis in Dravidian languages as part of the third Workshop on Speech and Language Technologies for Dravidian Languages at RANLP 2023. This shared task provides a platform for researchers worldwide to submit their models on two crucial social media data analysis problems in Dravidian languages - abusive language detection and sentiment analysis. Abusive language detection identifies social media content with abusive information, whereas sentiment analysis refers to the problem of determining the sentiments expressed in a text. This task aims to build models for detecting abusive content and analyzing fine-grained sentiment from multimodal data in Tamil and Malayalam. The multimodal data consists of three modalities - video, audio and text. The datasets for both tasks were prepared by collecting videos from YouTube. Sixty teams participated in both tasks. However, only two teams submitted their results. The submissions were evaluated using macro F1-score.

## introduction multimodal social media data analyses the insights from social media data that include multiple modalities such as text, audio, and videos. conventional social media data involves text data such as tweets, facebook posts, and youtube comments, and social media data analysis mainly focuses on processing text data to obtain valuable information. however, multimodal data analysis considers the diverse content shared on various social media platforms @xcite . analyzing the data con-taining multiple modalities can give a more comprehensive understanding of user behavior, opinions, and trends. the features extracted from video and audio data can be considered additional information to enhance the text features of input data. the facial expressions from the video, the pitch, and the tone of the audio signal can help to refine the features for identifying different opinions and expressions more effectively. multimodal social media data analysis combines approaches from different domains, including computer vision (cv), speech processing, and natural language processing (nlp), to process and detect different categories of data. the shared task on multimodal abusive language detection and sentiment analysis in dravidian languages at dravidianlangtech, organized as part of recent advancements in natural language processing 2023, has two subtasks -multimodal abusive language detection in tamil and multimodal sentiment analysis in tamil and malayalam. the main objective of the shared task is to encourage researchers around the globe to participate and submit their approaches and results, with the motivation to support the research in resource-poor languages like tamil and malayalam. sentiment analysis (opinion mining) @xcite , @xcite , @xcite identifies sentiments or opinions by analyzing the underlying subjective information in the text data. generally, sentiment analysis involves understanding the emotional tone present in a text to classify it as positive, negative, neutral, or more fine-grained categories. this analysis utilizes nlp embedding algorithms to represent text as a feature vector and classify the text into different categories using machine learning (ml) or deep learning (dl) algorithms. the multimodal sentiment analysis from video data involves the analysis of video or image sequence analysis and speech analysis in addition to the text analysis to identify the sentiments expressed in the data. in this task, the data are in tamil and malayalam, two low-resource languages. both languages are morphologically rich and agglutinative, which makes the analysis more complex. in addition, the codemixing property, which usually occurs in social media data, increases the complexity as it covers words from multiple languages. abusive language detection from social media data is identifying comments with abusive content @xcite , @xcite , @xcite , @xcite , @xcite . generally, in text-based analysis, the algorithms look for the words or phrases (sequence of words) or similar words/phrases that define the abusiveness of the input text. multimodal abusive language detection includes detecting videos containing abusive information by analyzing video, speech, and text data. this subtask is primarily focused on building models for multimodal data in tamil. this paper summarises the shared task on multimodal abusive language detection in tamil and multimodal sentiment analysis in tamil and malay-alam. besides, this paper discusses the findings from the models submitted to the two subtasks mentioned above. the shared task was hosted on codalab foot_0 . we shared the training data and validation data with all the registered participants for building their models. later, the test data without labels were shared to make predictions using the models built. sixty teams registered for both subtasks. however, only two teams submitted their results. the structure of the paper is as follows: section 2 discusses the research papers published in the domains of multimodal abusive language detection and multimodal sentiment analysis, section 3 describes the shared tasks, followed by section 4, which summarizes the systems submitted by participating teams. the paper concludes in section 5. ## related works people post messages and comments on various social media platforms in their mother tongue or codemixed languages. hence, the machine learning models built with monolingual datasets are unsuitable for identifying abusive language or analyzing sentiments from code-mixed languages. however, researchers are now progressing to develop systems using code-mixed datasets. as stated above, the data collection and annotation process is one key challenge. @xcite @xcite released a few dravidian language datasets for offensive language and hate speech detection. many models have been proposed using the above-cited datasets @xcite @xcite @xcite . the authors (singh and bhattacharyya, 2020) have employed an ensemble of multilingual bert models to detect hate speech and offensive content for dravidian languages. they have achieved an f-score of 0.95 for hate speech and offensive content detection tasks. in youtube comments in malayalam language(code-mixed, script-mixed). for hate speech and offensive content detection tasks for youtube or twitter datasets in malayalam (codemixed: tanglish and manglish), f-scores of 0.86 and 0.72 were achieved, respectively. @xcite experimented with multinomial naive bayes, support vector machines and random for-est methods to identify the offensive comments in code-mixed malayalam youtube comments. they have also used cross-lingual contextual word embeddings and transfer learning models to predict hate and offensive speech in malayalam data. a weighted average f1-score of 0.89 was achieved. an ensemble of multilingual transformer networks like xlmroberta has been proposed for offensive speech detection in code-mixed and romanised variants of three dravidian languages-tamil, malayalam, and kannada @xcite . two dl frameworks, namely cnn and bi-lstm @xcite , have been used in parallel to extract the contextual features from the text. these features were concatenated and presented to a fully connected layer for final prediction. they also used conventional machine learning approaches, such as naive bayes, random forests, support vector machine and transformer-based models to predict content from dravidian code-mixed scripts. @xcite explored various transformer models to detect the offensive language in social media posts in tamil, malayalam, and kannada. they achieved f1-scores of 0.9603 and 0.7895 on the malayalam and tamil datasets by using the ulmfit model, respectively. for the kannada dataset, they obtained an f1-score of 0.7277 by using the distilmbert model. two multi-task learning approaches @xcite ) was developed to identify the offensive language for kannada, malayalam, and tamil and analyse the sentiments. the model obtained a weighted f1-score of (59% and 70%), (66.8% and 90.5%), and (62.1% and 75.3%) for malayalam, kannada, and tamil on sentiment analysis and offensive language identification tasks, respectively. an offensive content classification model @xcite was built with transformer-based models, namely bert and roberta, for dravidian code-mixed languages like kannada, malayalam and tamil. the authors achieved a weighted average f1-scores of 0.72, 0.77, and 0.97 for kannada-english, tamil-english, and malayalam-english datasets. the authors @xcite developed datasets using the youtube comments of three dravidian languages, namely malayalam-english (20,000), tamil-english (44,000), and kannada-english (7000) for sentiment analysis and offensive language identification tasks. @xcite used machine learning and transformer-based models for offensive language identification. the authors represented the sentences using character n-gram and pre-trained word embedding. the f1-scores were 0.95 and 0.71 for malayalam and tamil, respectively. @xcite @xcite @xcite have used transformer-based models, such as bert, roberta and muril for offensive language identification task for dravidian languages. besides, research articles @xcite have shown the performances of offensive language detection from code-mixed languages using machine learning models such as logistic regression, k-nearest neighbour, support vector machine, decision trees and random forests. @xcite @xcite used deep learning techniques for offensive language identification task. all the above-discussed articles cover various methods and evaluations for offensive language identification and hate speech detection. they provide insights into this research's ensemble strategies and deep learning models. nevertheless, research has to progress deeper and wider to develop a robust model for hate speech and offensive language identification tasks using dravidian languages. even though text-based datasets and models are available for abusive language detection and sentiment analysis in dravidian languages, research on multimodal datasets is yet to kickstart. ## task description this section discusses the two subtasks -multimodal abusive language detection in tamil and multimodal sentiment analysis in tamil and malayalam, including the dataset used. the subtask, "multimodal abusive language detection in tamil", aims to encourage researchers to develop ai/machine learning/deep learning models for classifying video data posted on youtube into abusive and non-abusive categories. multimodal sentiment analysis in tamil and malayalam" subtask has two tasks -one in tamil and another in malayalam. in both tasks, the objective is to develop ai/machine learning/deep learning models for classifying video data into highly positive, positive, neutral, negative, and highly negative categories. in all the tasks mentioned above, video, audio, and text data were provided, and the participants were free to use any combination of modalities to build their models. the dataset used for conducting both tasks is insufficient for training a machine learning or deep learning model. however, the availability of pretrained models can help the participants to resolve this problem by generating meaningful feature representations. ## system description we received two submissions for both subtasks. each team was allowed to submit a maximum of three runs. the run with the highest macro f1 score was considered for preparing the rank list. the descriptions of the systems submitted for the shared tasks are given below. ## conclusion this paper reports the findings of the shared task on multimodal abusive language detection in tamil