Spartans@ LT - EDI - EACL 2021: Inclusive Speech Detection using Pretrained Language Models (2021)

Abstract: We describe our system that ranked first in Hope Speech Detection (HSD) shared task and fourth in Offensive Language Identification (OLI) shared task, both in Tamil language. The goal of HSD and OLI is to identify if a code-mixed comment or post contains hope speech or offensive content respectively. We pre-train a transformer-based model RoBERTa using synthetically generated code-mixed data and use it in an ensemble along with their pre-trained ULMFiT model available from iNLTK.

## introduction language has the ability to build relationships and forge connections but it is equally liable for creating barriers and impacting someone's sense of belonging. the language used on the internet has an impact on people across the globe. it is important to build language technology which makes everyone feel valued and included. we make contributions to this field by competing in two shared tasks: hope is considered significant for the well-being, recuperation and restoration of human life by health professionals. hope speech reflects the belief that one can discover pathways to one's desired objectives and become motivated to utilize those pathways @xcite . the goal of hsd task is to identify whether a youtube comment contains hope speech or not. the datasets are available in english, code-mixed tamil-english and malayalam-english. oli task intends to identify offensive language content in datasets comprising of comments/posts in code-mixed tamil-english, malayalam-english and kannada-english which are collected from social media. both datasets have been annotated at a comment level wherein a comment could comprise of more than one sentence but on average it has a single sentence. our work is an extension of the work done in (arora, 2020a) as we use their synthetic codemixed dataset for tamil and ulmfit model trained on that dataset. we pre-train a transformer based model roberta @xcite ) from scratch on code-mixed data and build an ensemble using ulmfit and roberta to achieve: we review some related work from literature before explaining the details of our approach and the results. all experiments described in the paper can be reproduced using the source code available on github foot_0 . ## related work as noted on the lt-edi 2021's website foot_1 , this is the first shared task on hsd. some work has been previously done for hsd @xcite , ## methodology in this section we look at classification datasets, discuss details of roberta and ulmfit models and the classifiers which are trained on top of these language models. ## experiment and results in this section we discuss details and results of our baseline model, classification models and ensemble strategy. ## conclusion and future work in this paper we present roberta language model for code-mixed tamil which we pre-trained from scratch. using transfer learning we fine-tune roberta and ulmfit language models on downstream tasks of oli and hsd. we got rank 4 in the former task using an ensemble of classifiers trained on roberta and ulmfit and rank 1 in the latter task using classifier based on ulmfit. in future research we will explore other transformer architectures like bert @xcite , t5 @xcite , xlm @xcite ). we will work on improving code-mixed data generation strategies. we plan to create a dataset using a combination of native tamil sentences, their transliterations and translations in english.