K austubh S hared T ask@ LT - EDI 2023: Homophobia-Transphobia Detection in Social Media Comments with NLPAUG -driven Data Augmentation (2023)

Abstract: Our research in Natural Language Processing (NLP) aims to detect hate speech comments specifically targeted at the LGBTQ+ community within the YouTube platform shared task conducted by LTEDI workshop. The dataset provided by the organizers exhibited a high degree of class imbalance, and to mitigate this, we employed NLPAUG, a data augmentation library. We employed several classification methods and reported the results using recall, precision, and F1-score metrics. The classification models discussed in this paper include a Bidirectional Long Short-Term Memory (BiLSTM) model trained with Word2Vec embeddings, a BiLSTM model trained with Twitter GloVe embeddings, transformer models such as BERT, DistiBERT, RoBERTa, and XLM-RoBERTa, all of which were trained and fine-tuned. We achieved a weighted F1-score of 0.699 on the test data and secured fifth place in task B with 7 classes for the English language.

## introduction the term "hate speech" refers to a specific style of offensive language that uses generalizations and stereotypes to convey an ideology of hatred @xcite . many people agree on the definition of "hate speech" as any kind of expression that targets an individual or group because of their race, color, ethnicity, gender, sexual orientation, nationality, religion, or other characteristic @xcite @xcite . the development of user-generated content online, particularly on social media platforms, has contributed to an increase in the amount of hate speech that is being distributed @xcite @xcite . over the past several years, there has been a rise in the amount of hate speech that can be found online, which has led to an increase in interest in the process of automating its detection @xcite . one such form of hate speech is homophobic or transphobic comments, which is hate targeted towards lgbtq+ peoples @xcite . the procedure of transphobia and homophobia detection entails discerning and isolating anti-lgbtq+ content within a given corpus. hate speech includes both homophobic and transphobic words, both of which are harmful to the lgbtq+ community @xcite . in our research, we addressed the issue of class imbalance in our dataset by employing data augmentation techniques using nlpaug, a python library specifically designed for augmenting text data. this approach effectively mitigated the degree of class imbalance. subsequently, we trained our models using various architectures including bilstm @xcite with pre-trained word2vec @xcite embeddings and twitter glove @xcite embeddings, as well as bert @xcite , distilbert @xcite , roberta @xcite , and xlm-roberta @xcite , while finetuning them. among these models, the best performance was achieved by the bilstm + word2vec model, which yielded a weighted f1-score of 0.56 on the validation dataset and 0.699 on the test dataset, placing it in the fifth rank in english. garcía-díaz et al. (2022) used a knowledge integration technique to train a neural network that effectively merges multiple feature sets. these include sentence embeddings in context and out of context, as well as linguistic features retrieved using a technique created by their research group. they got seventh, third, and second rank in english, tamil, and tamil-english respectively. following the implementation of sampling techniques to correct the data imbalance, feature extraction was conducted using a count vectorizer, tf-idf, and a variety of classifiers. among other techniques, svm classifiers, word embeddings, and bert-based transformers were utilized by @xcite . for the vectorization of remarks, tf-idf has been combined with various bigram models, and support vector machines was used to create the model by @xcite . @xcite utilized a collection of transformer-based models to construct a classifier and their system placed second for english, eighth for tamil, and tenth for tamil-english. @xcite used data augmentation and ensemble modeling along with different large language models (bert, roberta, and hatebert) to finetune, and the weighted majority vote was applied to their predictions. her proposed model received scores of 0.48 and 0.94 for the macro and weighted f1 scores, placing it in third place in english. ## dataset the training dataset comprised 3,164 data, while the validation dataset contained 999 data, with both datasets featuring a single column for text and another column for associated labels. the test dataset encompassed 990 data, with the objective to predict the corresponding labels. during preprocessing, the validation dataset underwent cleaning, resulting in a reduced size of 792 data. the validation dataset underwent cleaning because some data points did not have the output so we tried to remove those text data which didn't have its output labels @xcite . the number of labels for each class in the training dataset and validation dataset are given in ## methodology in order to address the issue of high-class imbalance present in the dataset, we applied data augmentation techniques using the nlpaug @xcite library in python. by augmenting the dataset, we aimed to ensure that the text inputs used for training the model would be diverse and representative, minimizing the risk of creating biases towards any specific label during prediction. the nlpaug used for augmentation, augmented each text around 7-10 times similarly the label size of the transphobic derogation in the original dataset consisted of only 1 label so we can augment it to only 12 sentences. further experimental analysis and the detailed methodology for conducting these experiments are elaborated in the subsequent subsections of this paper. ## future works in light of the suboptimal performance exhibited by transformers in this context, our forthcoming research will focus on refining their effectiveness through targeted fine-tuning strategies. specifically, we intend to explore the efficacy of diverse optimizers such as randomized search and keras optimizers to enhance the model's capabilities. additionally, we would aim to incorporate sentence augmentation techniques utilizing established libraries like nlpaug. furthermore, the integration of smote (synthetic minority over-sampling technique) would be explored to introduce text data diversity.