IIITT @ LT - EDI - EACL 2021-Hope Speech Detection: There is always hope in Transformers (2021)

Abstract: In a world with serious challenges like climate change, religious and political conflicts, global pandemics, terrorism, and racial discrimination, an internet full of hate speech, abusive and offensive content is the last thing we desire for. In this paper, we work to identify and promote positive and supportive content on these platforms. We work with several transformer-based models to classify social media comments as hope speech or not hope speech in English, Malayalam, and Tamil languages. This paper portrays our work for the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion at LT-EDI 2021- EACL 2021. The codes for our best submission can be viewed.

## introduction social media has inherently changed the way people interact and carry on with their everyday lives as people using the internet @xcite . due to the vast amount of data being available on social media applications such as youtube, facebook, an twitter it has resulted in people stating their opinions in the form of comments that could imply hate or negative sentiment towards an individual or a community @xcite . this results in people feeling hostile about certain posts and thus feeling very hurt @xcite . being a free platform, social media runs on user-generated content. with people from multifarious backgrounds present, it creates a rich social structure @xcite and has become an exceptional source of information. it has laid it's roots so deeply into the lives of people that they count on it for their every need. regardless, this tends to mislead people in search of credible information. certain individuals or ethnic groups also fall prey to people utilizing these platforms to foster destructive or harmful behaviour which is a common scenario in cyberbullying @xcite . the earliest inscription in india dated from 580 bce was the tamil inscription in pottery and then, the asoka inscription in prakrit, greek and aramaic dating from 260 bce. thunchaththu ramanujan ezhuthachan split malayalam from tamil after the 15th century ce by using pallava grantha script to write religious texts. pallava grantha was used in south india to write sanskrit and foreign words in tamil literature. modern tamil and malayalam have their own script. however, people use the latin script to write on social media @xcite @xcite . the automatic detection of hateful, offensive, and unwanted language related to events and subjects on gender, religion, race or ethnicity in social media posts is very much necessary @xcite . such harmful content could spread, stimulate, and vindicate hatred, outrage, and prejudice against the targeted users. removing such comments was never an option as it suppresses the freedom of speech of the user and it is highly unlikely to stop the person from posting more. in fact, he/she/they would be prompted to post more of such comments foot_1 @xcite . this brings us to our goal to spread positivism and hope and identify such posts to strengthen an openminded, tolerant, and unprejudiced society. text language label god gave us a choice my choice is to love, i would die for that kid english hope the democrats are.backed powerful rich people like soros english not hope este psic ã"pata mas ã"n luciferiano es hombre trans english not english neega podara vedio nalla iruku ana subtitle vainthuchu ahh yella language papaga tamil hope avan matum enkita maatunan... avana kolla paniduven tamil not hope i can't uninstall my pubg tamil not tamil ooororutharum avarude ishtam pole jeevikatte . k. malayalam hope etraem aduthu nilkallae arunae malayalam not hope phoenix contact me give you're mail i'd i hope i can support you sure! malayalam not malayalam hope is support, reassurance or any kind of positive reinforcement at the time of crisis (chakravarthi, 2020a). @xcite identifies the need for the automatic detection of content that can eliminate hostility and bring about a sense of hope during times of wrangling and brink of a war between nations. there have also been works to identify hate speech in multilingual @xcite and code-mixed data in tamil, malayalam, and kannada language @xcite . however, there have been very fewer works in hope speech detection for indian languages. ## dataset the dataset is provided by (chakravarthi, 2020a) (chakravarthi and muralidaran, 2021) and contains 59,354 comments from the famous online video sharing platform youtube out of which 28,451 are in english, 20,198 in tamil, and 10,705 comments are in malayalam ( subjects like hope speech might raise confusions and disagreements between annotators belonging to different groups. the dataset was annotated by a minimum of three annotators and the inter-annotator agreement was determined using krippendorff's alpha @xcite . refer ## experiment setup in this section, we give a detailed explanation of the experimental conditions upon which the models are developed. 4.1 architecture 4.1.1 dense the dense layers used in cnn (convolutional neural networks) connects all layers in the next layer with each other in a feed-forward fashion @xcite . though they have the same formulae as the linear layers i.e. wx+b, the output is passed through an activation function which is a non-linear function. we implemented our models with 2 dense layers, rectified linear units (relu) (agarap, 2019) as the activation function and dropout of 0.4. ## results in this section, we have compared the f1-scores of our transformer-based models to successfully classify social media comments/posts into hope speech or not hope speech and detect the usage of other languages if any. we have tabulated the weighted average f1-scores of our various models for validation and test dataset for english, malayalam and tamil languages in tables 5, 6 and 7 respectively. similar transformer-based model trained on multilingual data was used to classify malayalam and tamil datasets. models like multilingual bert, xlm-roberta @xcite , muril, indicbert foot_9 and distilbert multilingual with both bilstm and dense architectures. mbert (multilingual bert) uncased with bil-stm concatenated to it outperformed the other models for the malayalam validation dataset and continued its dominance for the test data as well. the data distribution for the tamil dataset seemed a bit balanced with an approximate ratio of 4:5 between hope and not-hope. mbert cased with bilstm architecture appeared to be the best model with an f1-score of 0.6183 for validation but dropped drastically by 8% for the test data. we witnessed a considerable fall in the scores of other models like mbert and xlm-roberta with linear layers of up to 15%. multilingual comments experience an enor-mous variety of text as people tend to write in code-mixed data and other non-native scripts which are inclined to be mispredicted. a variation in the concentration of such comments between train, validation and test can result in a fluctuation in the test results. the precision, recall and f1-scores of characterbert, mbertuncased, and mbert-cased are tabulated under english, malayalam, and tamil respectively, as shown in ## conclusion during these unprecedented times, there is a need to detect positive, enjoyable content on social media in order to help people who are combating depression, anxiety, melancholy, etc. this paper presents several methodologies that can detect hope in social media comments. we have traversed through transfer learning of several stateof-the-art transformer models for languages such as english, tamil, and malayalam. due to its superior fine-tuning method, ulmfit achieves an f1-score of 0.9356 on english data. we observe that mbert achieves 0.8545 on malayalam test set and distilmbert achieves 0.5926 weighted f1-score on tamil test set.