C om MA @ ICON : Multilingual Gender Biased and Communal Language Identification Task at ICON -2021 (2021)

Abstract: This paper presents the findings of the ICON-2021 shared task on Multilingual Gender Biased and Communal Language Identification, which aims to identify aggression, gender bias, and communal bias in data presented in four languages: Meitei, Bangla, Hindi and English. The participants were presented the option of approaching the task as three separate classification tasks or a multi-label classification task or a structured classification task. If approached as three separate classification tasks, the task includes three sub-tasks: aggression identification (sub-task A), gender bias identification (sub-task B), and communal bias identification (sub-task C). For this task, the participating teams were provided with a total dataset of approximately 12,000, with 3,000 comments across each of the four languages, sourced from popular social media sites such as YouTube, Twitter, Facebook and Telegram and the the three labels presented as a single tuple. For the test systems, approximately 1,000 comments were provided in each language for every sub-task. We attracted a total of 54 registrations in the task, out of which 11 teams submitted their test runs. The best system obtained an overall instance-F1 of 0.371 in the multilingual test set (it was simply a combined test set of the instances in each individual language). In the individual sub-tasks, the best micro f1 scores are 0.539, 0.767 and 0.834 respectively for each of the sub-task A, B and C. The best overall, averaged micro f1 is 0.713. The results show that while systems have managed to perform reasonably well in individual sub-tasks, especially gender bias and communal bias tasks, it is substantially more difficult to do a 3-class classification of aggression level and even more difficult to build a system that correctly classifies everything right. It is only in slightly over 1/3 of the instances that most of the systems predicted the correct class across the board, despite the fact that there was a significant overlap across the three sub-tasks.

## introduction the global reach of digital technology has resulted in the spread of social media applications to every section of society, making it a major medium of interaction for all kinds of people across the globe. social media sites have, as a result, become significant documents of human discourse for the digital age. social media discourse covers a broad spectrum and can be culturally and socio-politically specific to the region and people who engage in it, while also having a common grammar of form and content which have adapted to suit the platforms they appear in. a prime feature of social media discourse that has gained a lot of traction in the last few years is hate speech and aggression rooted in bias and prejudice. it manifests in the form of trolling, cyberbullying, flaming, and so on, and can have real-life consequences that are harmful, dangerous, and sometimes even fatal @xcite . the comma project aims to limit the negative effects of such comments on social media sites by developing a system that is trained to identify and isolate comments from social media platforms that display aggression and bias towards the target's gender and religious identities and beliefs. as part of our efforts in the project, we present this novel multi-label classification task to the research community, in which each sample will be required to be classified as aggressive, gender biased or communally charged. we expect that the task will be interesting for researchers working in the different related areas of hate speech, offensive language, abusive language as well more generally in text classification. ## related work automatically identifying the various forms of abusive language online has been studied from different angles. examples include trolling @xcite @xcite , flaming/insults @xcite , radicalization @xcite , racism @xcite , misogyny @xcite @xcite @xcite , online aggression @xcite , cyberbullying @xcite , hate speech @xcite @xcite @xcite and offensive language @xcite . the terms used in the literature have overlapping properties as discussed in @xcite and @xcite . most related studies focus on english, but a significant amount of work has been carried out for other languages too. this includes languages such as arabic @xcite , german @xcite , greek @xcite , hindi @xcite . the field has also seen a rapid development and availability of multiple datasets in multiple languages via various shared tasks and competitions, this shared task is one of many shared tasks that are being organised in this area, which include @xcite @xcite @xcite @xcite . among these, one of the most popular tasks, of-fenseval series of tasks @xcite , focused on offensive language identification and featured three sub-tasks: offensive language identification, offensive type identification, and offense target identification building on the annotation model introduced in the olid dataset @xcite for english. this multiple sub-task model has been adopted by other shared tasks such as germeval for german @xcite , hasoc for english, german, and hindi @xcite , and hateval for english and spanish @xcite . the tasks most similar to the current one were the trac -1 and trac -2 shared tasks. trac -1 shared task on aggression identification @xcite ) was hosted at the trac workshop at coling 2018. it included english and hindi data from facebook and twitter. it consisted of a three-way classification task with posts labelled as overtly aggressive, covertly aggressive, and nonaggressive. trac -2 @xcite featured data from 3 languages -bangla, hindi and ebglish -and included an additional sub-task of misogyny identification. the present task has been conceptualised as an extension of the trac-2 shared task, with more languages and an addition sub-task. moreover, it is now also reformulated as a structured prediction task, along with three separate text classification tasks, to encourage teams towards leveraging the benefits of a multi-task setup in a largely overlapping setup. ## task schedule and setup participants for the present shared task were allowed to participate in one of the four languages -meitei, bangla, hindi, or multilingual -or all of them but they were required to submit predictions for all three subtasks @xcite . the english data is not provided separately and is included in the data of all the languages. registered participants got dataset (training, development and test set) for training and evaluation in all languages through the codalab platform foot_0 . for the task, the participants were given around 4 weeks to experiment and develop the systems. after 4 weeks of releasing the train and development sets, the test set was released, after which the participants had 6 days to test and upload their systems. the entire timeline and schedule of the shared task is given in in the evaluation phase, each team was permitted to submit up to 5 systems and their best run was included in the final ranking presented in this paper. ## dataset we provided a multilingual dataset with a total of over 15,000 samples for training, development and testing in four languages: meitei, bangla, hindi, and english. the dataset was marked at three levels: aggression, gender bias, and communal bias. each level was represented in the form of an individual sub-task: ## participants and approaches a total of 54 teams registered for this shared task, with most of the teams registering to participate in all the languages. by design, all the teams were required to participate in all the three tracks. finally a total of 11 teams submitted their systems -out of these, 8 teams have been included in the official rankings while the other 3 are not because of delayed submission on their part -however they were also evaluated and are discussed here. all the 11 teams that submitted their system were invited to submit the system description paper, describing the their models and experiments conducted by them. the name of the participating teams and the language they participated in are given in we give a brief description of the approaches used by each team for building their system. a detailed description of the approaches could be found in the paper submitted by each team. we give a brief summary of each team's system below - ## error analysis we carried out an overall analysis of the errors generated by all the systems submitted for the task. this was done with an aim to understand the most difficult instances to classify. in this error analysis, we have analysed only those instances which have been classified wrongly by 'all' the models for subtask a and those which have been clasified wrongly by at least ' foot_2 4 ' of all models in case of sub-task b and c 3 in all languages. a summary of the errors generated by the systems on the test data in all the languages have been presented below under "error types". language wise error counts and error type counts in all sub-tasks are given in tables 7 and 8 and figure 1 . we identified the recurring patterns that generate these errors and classified them as follows: context overgeneralization out-of-vocab lack of sufficient features 0% 25% 50% 75% 100% task a multi task b multi task c multi • context: contextual errors occur when there is a mismatch between the gold and predicted labels of a comment based on whether or not the annotator or the system has taken into account the discursive context in which the comment exists. such a context can include the contents of the video or post under which the comments are written, the other comments that are in conversation with or appear alongside the given comment, and the socio-political context in which certain content and comments find expression. the comments that have generated context based errors in this shared task include sarcastic or satirical comments, ambiguous comments (that can be legitimately labelled with more than one tag), and replies to previous comments (in the sense that they could be correctly classified only by ## closing remarks in this paper, we have presented the results of the shared task on automatic identification of aggressive language, gender bias and communal polarisation. the results show that while it is relatively easier to get prediction on one of these categories right, it is still a very difficult task to predict all of these right for a single instance -the best team managed to get an instance f1 of only 0.371. however at the same time, we also see that the best result across all models and all teams is attained by a model that is jointly trained for all the sub-tasks and all the languages -this shows the value of multitask and multilingual learning in low-resource situations. the second major takeaway related to the models is that ensemble of well-tuned linear classifiers are also useful for tasks like these and we see that one of the systems in top-3 is an ensemble system. in other instances as well, ensembles have proved to be better than or equivalent to the transformers-based systems. in terms of the model performance (and also reliability of the dataset), a comprehensive error analysis of the models submitted for the task show that a huge majority of the errors made by all the model relates to the generalisability of the models, manifested in terms of overfitting for certain linguistic features and inability of the models to perform well on data outside of the training set domain. this could be attributed to two possible reasons -1. lack of sufficient datapoints for system to generalise well -this could improved by augmenting the dataset with more instances. in addition to this, the other most common source of error is the lack of contextual knowledge in the way dataset is presented and the way models are trained. this could be improved only by providing explicit contextual information in the dataset and also for models to take into consideration those information. we plan to make this available in the next version of the dataset.