Overview of Second Shared Task on Homophobia and Transphobia Detection in Social Media Comments (2023)

Abstract: We present an overview of the second shared task on homophobia/transphobia Detection in social media comments. Given a comment, a system must predict whether or not it contains any form of homophobia/transphobia. The shared task included five languages: English, Spanish, Tamil, Hindi, and Malayalam. The data was given for two tasks. Task A was given three labels, and Task B fine-grained seven labels. In total, 75 teams enrolled for the shared task in Codalab. For task A, 12 teams submitted systems for English, eight teams for Tamil, eight teams for Spanish, and seven teams for Hindi. For task B, nine teams submitted for English, 7 teams for Tamil, 6 teams for Malayalam. We present and analyze all submissions in this paper.

## introduction a victim, an aggressor, and bully-victims are all necessary components of the aggressive behavior known as bullying @xcite . students, parents, teachers, and administrators all share a considerable concern with the phenomenon of bullying that is motivated by homophobia @xcite @xcite . the unfavorable views, attitudes, prejudices, and behaviors that are held towards sexual minorities are what are referred to as homophobia @xcite . homophobia is the underlying mentality that plays a contributing role in the practice of discrimination against lgbtq+ vulnerable individuals @xcite . the fact that the most prevalent definition of homophobia is "an attitude of hostility toward male or female lgbtqi+ vulnerable individuals" implies that this idea is rather narrow and has a tendency to individualize the process of discrimination and rejection @xcite . the internet is a tool that lgbtqi+ vulnerable young individuals in re-gional, remote, and rural areas use to overcome isolation and construct relationships that extend beyond the physical limitations of a geographic area by commenting on youtube videos or reaching out via twitter or other means in social media @xcite @xcite . since social media is used by vulnerable individuals, it should be without homophobic/transphobic bullying or other hate speech against lgbtq+ vulnerable individuals @xcite @xcite . to tackle homophobia and transphobia in social media, @xcite introduced a new dataset in english, tamil, and tamil-english. @xcite conducted new shared tasks in tamil, english, and tamil-english (code-mixed) languages. it received 10 tamil systems, 13 english systems, and 11 tamil-english systems. the average macro f1-score for the top systems for tamil, english, and tamil-english was 0.570, 0.877, and 0.610, respectively. @xcite conducted a shared task on sentiment analysis and homophobia detection; in that task, new language data, malayalam, was added. in our task foot_0 , we conducted two sub-tasks: task a and task b. we included english, tamil, spanish, malayalam, and hindi for task a and english, tamil, and malayalam for task b. overall submission of eleven teams that participated in task a and eight teams that participated in task b. the weighted f1 scores of top-performing models for these languages are 0.888, 0.997, 0.979, 0.969, and 0.949. for task b, we included english, tamil, and malayalam; the top-performing model scored with weighted f1 scores of 0.822, 0.884, and 0.865. we divided the task into two subtasks: a and b. ## methods of participants the team "teamplusone" submitted a system for task a and b with an english dataset. they used a pre-trained model called bert(bidirectional encoder representations from transformers). they used the default parameter setting for training. with this, they were able to achieve the weighted f1 score of 0.9692868 in task a and 0.8221297 in task b. the "supernova" @xcite team used term frequency-inverse document frequency (tf-idf) for classifying both tasks. tf measures the frequency of a term within a document. since support vector machines(svms) are known for their ability to handle overfitting, this team used svm to classify both tasks. this team also claimed that svms can perform well even with relatively small training datasets and generalize effectively from limited examples, making them suitable for sentiment analysis applications in various domains. a team "ssntech2" @xcite ) classified task a. for this task, the team first preprocessed and cleaned the dataset and assigned token values to each category. they used the nltk module for preprocessing, such as stop word removal, lemmatizing and normalizing, and removing stop words. for the first test run, the team used the sgd classifier. it produced an accuracy of 0.93, an f1 average score of 0.38, and a weighted score of 0.92. for the second test run, the team used the svm classifier. it produced better results than the sgd classifier, with an accuracy of 0.94, an f1 average score of 0.42, and a weighted score of 0.94. svm is used for classifying the dataset in english under task a by the team named "tercet" @xcite . given a higher precision, f1 score, and weighted averages compared to the random forest, logistic regression, and naive bayes models, svm was a good fit for classifying the test datasets. the team did preprocess the text data, such as removing punctuation, emoticons, and stop words. to convert the text data into a form that is usable by the model, the team also used the tf-idf vectorizer algorithm. it utilized the extracted features in the svm classifier. they used the tf-idf vectorizer algorithm to convert the text data to the model understandable form. then svm classifier used the vectorized features for the classification. the "cordyceps" @xcite team classified both tasks using a weight-space ensembling technique. first, they trained a multilingual model on a dataset that included all the languages and then created finetuned models for each language. ultimately, for each language, they performed linear interpolation between the finetuned and multilingual models' weights. the resulting interpolated model is then used for inference. the selection of the linear interpolation parameter is based on a held-out validation set consisting of samples in the language of the finetuned model that were not encountered during training. the team also observed that weight-space ensembling enhances performance, particularly for low-resource languages. the most interesting aspect of this work is the novel application of weight-space ensembling on codemixed data, aiming to leverage the strengths of both multilingual and finetuned models for improved performance in analyzing mixed-language text. a custom pre-trained xlm-roberta transformer-based multilingual model has been developed by the team "cantnlp" @xcite . this team has pre-trained the language model with a random sample of 50,000 tweets (over 50 characters) for each language condition. for the language conditions with brahmic scripts (hindi, malayalam, and tamil), the team romanized a quarter of the text samples to simulate the team "deepblueai" finetuned xlm-roberta as the base model for classifying both tasks. this team has attempted mixing multiple language datasets at different proportions and performed cross-validation. the team "adsa nlp sys" used svm in conjunction with tf-idf vectorization for classifying the english comments under task b and used the adasyn sampling technique. in order to hypertune the model, the team has used tf-idf grid. the team claimed that adasyn, with tf-idf and grid search, can find the best model and parameters. the team "kaustubhsharedtask" @xcite participated in task b in the english dataset. due to class imbalance in task b's training dataset in the english language, they used nlpaug -a tool to augment the text data and reduce the degree of imbalance. in augmentation of the text data, they tried several parameters like synonym replacement, word insertion, and word substitution to get augmented sentences with the same meaning as the original sentence. they did text preprocessing and applied various transformers models with fine tuning and got the best results on the bilstm model trained on the word embeddings generated from word2vec. the "mucs" @xcite team has tried using mbert(multilingual bert) and resampling with bert to classify both tasks. for feature extraction, they used tf-idf. a gpt2 model has been used by the team "ju-dithjeyafreeda" (andrew, 2023) to finetune the training set for classifying both tasks. for using this model, the team substituted the comments in other languages with english letters. the team "vel" @xcite utilized the "muril-large-cased" model, which is a variant of the gpt-3.5 architecture developed by openai to classify the spanish comments under task a. in addition to using the "muril-large-cased" model, the team also employed machine learning techniques such as naive bayes (nb), support vector machines (svm), logistic regression (lr), decision trees (dt), and random forests (rf) with count vectorizers. ## results and discussion overall, we received a total of 10,7,4,7, and 7 submissions for english, tamil, spanish, malayalam, and hindi in task a. for task b, we received 8,7, and 6 submissions for english, tamil, and malayalam in task b. the tables 9,13,12,10 and 11 shows the rank list of all languages of task a, and the tables 14, 16, and 15 shows the rank list of all languages of task b. in task a, the model of the team "teamplusone" achieved the top-performing model in english and hindi language. they used bert pretrained model with the default setting for the training and achieved the weighted f1 score of 0.96928 and 0.97933, and the "cordyceps" model is the top-performing model in tamil, malayalam, and spanish languages. they used the weight space ensembling technique, improving the performance of analyzing the mixed language text. they achieved 0.94968, 0.99769, and 0.88831. the top-performing models in task b are the model developed by the team "teamplusone," ranked 1st in the english language. they used bert model for training with a default parameter setting. they achieved a weighted f1 of 0.82212. in tamil language, the "deepblueai" team's model got the 1st rank. they used the xlm-roberta base model and performed cross-validation by combining multiple language datasets in varied amounts, which gained the weighted f1 score of 0.88429. for malayalam, the "cantnlp" team developed the custom pre-trained xlm-roberta model with 50000 random tweets, and they also oversampled the training to tackle the class imbalance problem. this model achieved 0.86515. ## conclusion we presented the second shared task findings on homophobia/transphobia detection in social media comments in this publication. we got an extensive variety of entries that fulfilled the aims of the shared task. we expect that the shared task on homophobia/transphobia detection will have a long-term impact on the nlp discipline.