L eaning T ower@ LT - EDI - ACL 2022: When Hope and Hate Collide (2022)

Abstract: The 2022 edition of LT-EDI proposed two tasks in various languages. Task Hope Speech Detection required models for the automatic identification of hopeful comments for equality, diversity, and inclusion. Task Homophobia/Transphobia Detection focused on the identification of homophobic and transphobic comments. We targeted both tasks in English by using reinforced BERT-based approaches. Our core strategy aimed at exploiting the data available for each given task to augment the amount of supervised instances in the other. On the basis of an active learning process, we trained a model on the dataset for Task i and applied it to the dataset for Task j to iteratively integrate new silver data for Task i . Our official submissions to the shared task obtained a macro-averaged F _1 score of 0.53 for Hope Speech and 0.46 for Homo/Transphobia, placing our team in the third and fourth positions out of 11 and 12 participating teams respectively.

## introduction in recent years, many episodes of violence against homosexuals and transsexuals have been observed online (e.g., in youtube comments 1 ) and offline, which escalated into the death of 375 transgender people in 2021 alone. 2 most of the victims were black and latin women, especially sex workers, a fact that highlights the intersection between misogyny, racism, xenophobia and hate towards sex workers. that is why identifying such behaviours online is timely, as it can contribute to limiting the spread of hate. in this regard, two different tasks have been proposed in lt-edi in various languages: classify a youtube comment into homophobic, transphobic or non-anti-lgbt content @xcite . hope speech detection (task hope ) classify a youtube comment into hope speech or non-hope speech @xcite . we approach both tasks, addressing the english language only. 3 we experiment with two different approaches for task hope and four for task antilgbt . we aim at augmenting the data to cope with the heavy imbalance in the datasets. all models are built on top of bert @xcite . for task hope we implement a binary classifier which is our baseline, and we augment data through an active learning approach @xcite . for task antilgbt we implement a multi-class classifier as our baseline. then, we augment training data according to three approaches: the rest of the paper is structured as follows. section 2 provides an overview of definitions and related work in the field of abusive language detection, focusing in particular on homophobia, transphobia (and hope speech). section 3 explores the two datasets provided by the shared task. section 4 describes our models for both tasks and section 5 outlines the hyperparameters and preliminary experiments. section 6 presents and discusses our results. finally, section 7 draws conclusions. ## background the importance of the automatic detection of abusive language has increased together with the popularity of social media @xcite . the online discourse often has hateful and offensive connotations towards minorities. the exposure to hate speech can trigger polarization, isolation, depression, and other psychological trauma @xcite . becoming aware of this serious societal issue, online platforms have assumed the responsibility of examining and removing hateful posts @xcite . due to the continuous flow of large amounts of contents through social media, hatred is flagged through automatic methods along with human monitoring @xcite . in order to foster the development of automatic models for the identification of different kinds of hate speech, diverse supervised datasets and models have been developed. @xcite proposed a dataset with homophobic and transphobic contents from youtube, gathering comments from famous youtubers that raise awareness on the lgbt+ community and also from channels that report pranks and jokes about homosexuals and transsexuals. given the sensitivity of the topics covered in the videos, the comments posted can often have abusive, offensive or denigratory connotations towards the lgbt+ community. they found out that a combination of machine learning models, including random forests @xcite reinforced with bert embeddings @xcite , obtains the best result. hope speech, on the other hand, lies on the other end of the spectrum of digital rhetoric. in contrast to hateful comments, a hopeful discourse is characterized by a friendly tone and an intention to inspire, support, include, and encourage members of minorities, who are often subject to judgment, isolation, and suffering (chakravarthi, 2020). focusing on spotting hopeful rather than hateful contents offers a twist that seeks to produce a better online ecosystem by promoting rather than limiting comments and opinions. this angle was explored within the hope speech detection shared task (chakravarthi, 2020) on hopeedi, a multilingual collection of youtube comments. 4 according to chakravarthi and muralidaran (2021), the best approach for english train test hope speech 2,234 non-hope speech 23,347 - ## datasets here, we briefly describe the datasets for task antilgbt and task hope . task antilgbt the collection consists of comments of youtube videos that were annotated by lgbt+ community members. task hope ## systems overview in the following paragraphs, we first describe the active learning approach. we then present the specific strategies developed for task antilgbt and task hope respectively. for task antilgbt , we trained four alternative models to identify the best possible configuration: baseline, baseline augmented with tamil data translated to english, baseline augmented with hope speech data remapped as non-anti-lgbt content and baseline with augmented data from task hope through an active learning approach. for task hope , we trained two alternative models: the baseline and the active learning approach. cross-task data augmentation through active learning the two tasks at hand are related, as the labels of both datasets can be traced back to hateful and non-hateful instances. instances of homo/transphobic and hope speech messages can be remapped to their non-hope speech and non-anti-lgbt comments respectively. on the contrary, it is not always true that a non-hope speech instance is homo/transphobic and that a non-anti-lgbt content contains hope speech. therefore, given the small amount of training instances available for both task antilgbt and task hope , we aim to take advantage of both datasets proposing an approach to augment the training sets for each task. we first add the homo/transphobic and hope speech instances in bulk, and then we filter the uncertain ones, i.e., nonhope speech for task antilgbt and non-anti-lgbt content for task hope , through an active learning approach (hino, 2020) as follows. let d i and d j be the supervised datasets for both tasks. (i) train model m i on d i . (ii) predict the instances in d j with m i . (iii) rank the instances in d j according to the confidence of the prediction score returned by m i . (iv) transfer the top-k instances in d j as silver data to d i . this process is repeated until |d j | = ∅ and the final model for task i is then used to predict on the dev set for task i. specifically, we augment the dataset for task hope by adding in bulk homophobic and transphobic instances remapped to non-hope speech instances. we do the same for task antilgbt by adding in bulk hope speech instances to non-anti-lgbt content. then, we use an active learning approach to identify which non-anti-lgbt instances contain hope speech, and which non-hope speech instances contain homophobia/transphobia. in the end we integrate the identified instances (i.e., hope speech and homo/transphobic) in both datasets. figure 1 represents the approach for task hope . first, homophobic and transphobic instances from task antilgbt are added as non-hope speech. then, we feed non-anti-lgbt instances to the model trained on task hope dataset. those which are predicted as hope speech are integrated in the training set. we adopt the same approach for task antilgbt . ## experimental setup no preprocessing is applied to the text, other than applying the berttokenizer @xcite . we shuffle the training set and take 10% of the data for development, preserving the class distribution through stratified random sampling @xcite . in order to find the best hyperparameters to predict on the test set, we experimented with different batch sizes (4,8,16) for the baseline model, over an increasing number of epochs (4,6,8), testing on the development set. the combination that performed the best was a batch size of 16 over 4 epochs for both tasks, therefore we used those hyperparameters to train all models. in order to model variation f 1 bert baseline 0.76 bert active learning 0.77 as for the evaluation metrics, we stick to the official one: macro-averaged f 1 -measure for both tasks. since task antilgbt is a multi-class problem, we computed the weighted f 1 -measure when testing on the development set. ## results in this section, we present our results for both tasks. for task antilgbt we provide the results generated with the predictions of both development and test sets. for task hope , we present only the results on the development set. foot_4 6.1 performance on the development set task antilgbt task hope ## conclusions and future work this paper provided a description of our participating models to the lt-edi-acl2022 shared tasks on hope speech detection and homophobia/transphobia detection. we addressed the two problems together, by exploiting data available in one task to create silver data for the other task. for task antilgbt , our baseline outperforms all the other reinforced approaches which make use of external data when tested on the test set. 'for what concerns the active learning approach, it is likely that non-hope speech data do not contain homophobia or transphobia, contrary to what we expected, and therefore they do not contribute to increase the performance for task antilgbt , as shown by our experiments. for task hope the active learning approach outperforms the baseline in the development set by one unit only, and it achieves the same score as the baseline in the test set, concluding that the impact of transferring data from one task to the other is model variation f 1 prec rec bert baseline 0.53 0.53 0.53 bert active learning 0.53 0.53 0.53 iiitsurat (1) 0.55 0.56 0.54 mucic (1) 0.55 0.54 0.55 arguably (2) 0.54 0.55 0.54 in future work, we would like to test other transformer-based models to assess the impact of different pretraining techniques on the effectiveness of the active learning approach for these particular tasks. it would also be interesting to try different evaluation approaches for these tasks by exploring the fairness of classifiers @xcite , with respect to minority social identities, i.e., the different members of the lgbt+ community. specifically, we would like to investigate whether the classifiers contain unintended biases, e.g. towards specific sexual orientations, according to well-known metrics proposed to detect unfairness within toxicity detection @xcite .