IIITK @ LT - EDI - EACL 2021: Hope Speech Detection for Equality, Diversity, and Inclusion in T amil , M alayalam and E nglish (2021)

Abstract: This paper describes the IIITK’s team submissions to the hope speech detection for equality, diversity and inclusion in Dravidian languages shared task organized by LT-EDI 2021 workshop@EACL 2021. Our best configurations for the shared tasks achieve weighted F1 scores of 0.60 for Tamil, 0.83 for Malayalam, and 0.93 for English. We have secured ranks of 4, 3, 2 in Tamil, Malayalam and English respectively.

## introduction according to wikipedia hope is being defined as an optimistic state of mind that is based on the expectation of outcomes with respect to events and circumstances in one's life or the world at large. the hope speech detection shared task 2 organized by lt-edi aimed to detect hope speeches in the given corpus for english, tamil, and malayalam @xcite . the data set has been gathered from some social media remarks. we participated in this task given a social media remarks in hope speech, frameworks need to characterize if a post is hope speech or not. tamil (iso 639-3: tam) and malayalam (iso 639-3: tam) belong to same family. tamil was the first to be listed as a classical language of india, one of 22 scheduled languages in the constitution of india, also official language of tamil nadu, puducherry, singapore and sri lanka and is one of the world's longest-surviving classical languages 1 https://github.com/nikhil6041/ hopespeechdetection 2 https://competitions.codalab.org/ competitions/27653 @xcite @xcite . the oldest epigraphic documents discovered date from about the 6th century bc on pottery, rock edicts and hero blocks. over 55 percent of the epigraphic inscriptions discovered by the archaeological survey of india (about 55,000) are in the tamil language @xcite . a tamil prayer book in ancient tamil script called thambiran vanakkam was written by portuguese christian missionaries in 1578, thereby rendering tamil the first indian language to be printed and published @xcite . malayalam split from tamil during 16th century by thunchaththu ramanujan ezhuthachan until then it was west coast dialect of tamil @xcite . over time various methodologies are being proposed by the researchers throughout the natural language processing (nlp) community for building better textual analysis systems. solving the text classification problem has been improvised throughout by building better architectures and better representation techniques for texts. the community has also benefitted by borrowing the ideas from other domains like computer vision and incorporating those in these systems which have given promising results. initially, the models used to deal with the bag of words (bow) representations, then came the ideas of lemmatization and stemming, which helped in improving the representation techniques further. then around the early 2010s word embeddings were proposed by @xcite . the nlp domain has also observed many architectural innovations which have further pushed the performances to give state of the art (sota) results. some of them are the lstms @xcite , bilstms @xcite , grus @xcite and then the mighty transformers @xcite . the introduction of transformers changed the entire land-scape, and the models built upon the transformer architecture are consistently pushing the results on the glue @xcite benchmarks. there have been instances where the researchers have tried to incorporate the architectural innovations from different domains to nlp. in this paper, we have tried several architectures built upon the transformer architecture and have fine-tuned them on our task, details of which are being discussed in the later sections of the paper. ## related work hope speech detection is a relatively new field and an active area of research in the nlp domain. with the rise of the internet and the social media platforms, people from various places around the globe are now connected through these platforms which have given them a common place to express their views. these views can often be specifically targeted to a particular person or community that can convey either of a positive, neutral or negative emotion to the concerned person or community. this makes it an important aspect to have systems that can automatically classify these content and filter out the ones having a negative impact on the society. in other words, this also means that we have systems that explicitly detect positive content and help it stay in the social good system. as defined earlier, hope speech can also be considered a piece of text conveying a positive sentiment to the reader of it. one of the first works on hope speech detection is done by chakravarthi (2020a), @xcite , and @xcite . @xcite used the polyglot word embeddings to have clusters of texts that conveys similar sentiments and obtained promising results. hope speech detection can also be considered as the opposite task of hate speech detection. there has been a significant amount of work done for the hate speech detection task @xcite @xcite @xcite . it has even been a part of several conferences like semeval 3 as challenges. however, these conferences mainly focused on datasets which were constructed for resource abundant languages. however, in the mid-2020s several competitions have been organized which centred around these underresourced languages. to build a system that performs well on under-resourced languages like dra-3 https://semeval.github.io/ vidian languages, several researchers have developed systems that have given noticeable results on these tasks @xcite @xcite . hasoc-dravidian-codemix-fire2020 participants used traditional ml methods like naive bayes classifier, support vector machines (svms) and random forest along with the pretrained transformers models like xlm-roberta (xlmr) @xcite and bert @xcite for the offensive content identification in code-mixed datasets (tamil-english and malayalam-english). @xcite at hasoc-dravidian-codemix-fire2020 used ulmfit @xcite to pretrain on a synthetically generated code-mixed dataset and then fine-tuned it to the downstream tasks of text classification. the competition organizers have provided us with datasets (chakravarthi, 2020a) for three different languages tamil, malayalam and english. across each dataset we had three different classes hope speech, non hope speech and not lang where lang can be either of tamil, malayalam or english depending upon the dataset we are dealing with. the train, dev and test set distributions of the dataset are as shown in the ## methodology the analysis of the nature of texts has been one of the central tasks in nlp. textual analysis can be defined as a separation of the texts into different classes based upon the underlying meaning they convey @xcite . the nlp domain has observed many advancements for solving the textual analysis problem. however, it remains an unsolved problem because of the linguistic diversity worldwide and the difficulties in expressing texts to a suitable format for feeding it into the textual analysis systems @xcite . over time, various methods have been proposed for representation of texts, ranging from bag of words, tf-idf to word embeddings. the word embeddings were introduced with the word2vec model, which gives a vectorized representation for a word. after the introduction of the word2vec word embedding model, different word embedding techniques have been proposed throughout the nlp domain such as glove @xcite , doc2vec (le and mikolov, 2014), fasttext @xcite . using these different kinds of word representations, various different models have been proposed for solving the textual analysis problem. these models consisted of the primitive machine learning models like naive bayes (nb), logistic regression (lr), multinomial naive bayes (mnb), support vector machines (svms). apart from these models, various models based upon neural networks were also being used such as lstms, bidirectional lstms, grus. however, the current state of the art (sota) models are based upon the transformer architecture. there are numerous models built upon the transformer architecture which were being trained on large corpora of texts and are available for fine-tuning to different downstream tasks like textual classification, question answering. these models based upon the transformer architecture uses their tokenizers for the conversion of texts into embeddings which are based upon their own vocabularies. one major problem faced with these models built upon the transformer architecture is that they are only available for high resourced languages like english, german, and chinese. to use these models for under-resourced languages, the researchers came up with the idea of cross-lingual transfer learning, which means training a model on a high resourced language and then fine-tuning it on a downstream task. a separate benchmark known as xnli @xcite was being made to evaluate the model's performances across multiple languages. the competition organizers have used the svms, mnbs , decision trees and other machine learning models as the baseline models for the given datasets. so, we went with using the models built upon the transformer architecture while approaching the problem. we have used the hugging face foot_0 transformers library for our implementations and used the original versions of the models as well as their customized versions with different loss functions. we have used multilingual-cased bert (mbert-cased), xlm-roberta (xlmr), indicbert @xcite , bert-base-cased (bert-cased) and bertbase-uncased (bert-uncased) models for our implementations. pertaining to the large size of mbert-cased and xlmr models we have used their customized versions as well by freezing the original model and stacking a fully connected layer of 512 neurons with a final layer having the same number of neurons as the number of our output classes. with this customized versions, we have used two different loss functions the negative log likelihood (nll) loss function with class weights, and the sadice @xcite loss function both of which were used to handle the data imbalance in the datasets. a pictorial representation of our customized architecture can be seen in the figure 1 . out of all the models mentioned above, mbertcased, xlmr, and indicbert are multilingual models, and bert-cased and bert-uncased models are monolingual models. figure 1 : custom architecture : we defined our custom architecture apart from the original transformer models as being built upon the transformer models as the base unit. the output attention heads from the transformer layers are further connected to a 512 neuron fully connected (fc) layer which is finally connected to another fully connected layer having number of neurons same as the number of classes denoted by nc ## results and discussion we have tried several different combinations of models discussed in the section 4 across the datasets of each language and have reported our results on the development set and the test set in the the original versions of these models as well as their customized versions, were also being used for finetuning purposes over the datasets. attributing to the huge model size of mbert-cased and xlmr we have also tried out their customized versions with the nll loss and sadice loss functions. since the indicbert model is comparatively smaller than the other models, its original version was only being considered. apart from these multilingual models two different monolingual models for the english dataset were also considered. out of all the models, the original versions outperformed the customized versions of the models. for the tamil dataset xlmr, mbert-cased and indicbert gave similar results on the development dataset. however, mbert-cased gave comparatively better performances than xlmr and indicbert on the test dataset. for the malayalam dataset, xlmr, mbert-cased and indicbert had almost equivalent performances on the development dataset, but mbert-cased gave much better results on the test set. surprisingly, xlmr performed even worse than the indicbert model on the test set for malayalam. for the english dataset, apart from the xlmr, indicbert, mbert-cased the bert-cased and bert-uncased versions were also being tried and almost each model performed equivalently on the development dataset as well as the test datasets. the superior performance of mbert over the other two models can be attributed to the training strategy of mbert. it employs zeroshot cross-lingual model transfer, in which taskspecific annotations in one language are used to fine-tune the model for evaluation in another language. a brief explaination of the multilingual nature of mbert is being discussed in @xcite . on the other hand xlmr although being trained over much more data and having the same training strategy as @xcite was expected to perform better across the multilingual tasks but it hasn't. we hypothesize the reason behind the degradation in it's performance can be attributed to the code-mixed nature of our dataset in hand. since the xlmr model was being trained over the com-moncrawl data it could be possible that the data being utilised for the pretraining had very fewer instances of code-mixed data which thus leads to an overall inferior performance as compared to other models. the customized versions of these models were expected to address the skewness of the dataset but failed to do so. when inspecting these model's performances the reason for this performance degradation turned out to be the freezening of the base layers of the transformer models. the performance of these custom models can be further improved by having unfreezed layers which can further increase the performance of these models and can be considered for the future works on this task. ## conclusion we have presented the iiitk team's approach for the hope speech detection shared task organized by dravidianlangtech. our approach consisted of using the existing pretrained models and finetuning their original as well as the custom versions on the datasets. out of all the models, the mbertcased model gave the best results for the tamil and malayalam datasets as 0.60 and 0.83 weighted f1 scores. for the english dataset, mbert-cased and bert-cased gave exactly similar results of 0.93 weighted f1 scores.