VEL @ LT - EDI -2023: Automatic Detection of Hope Speech in B ulgarian Language using Embedding Techniques (2023)

Abstract: Many people may find motivation in their lives by spreading content on social media that is encouraging or hopeful. Creating an effective model that helps in accurately predicting the target class is a challenging task. The problem of Hope speech identification is dealt with in this work using machine learning and deep learning methods. This paper presents the description of the system submitted by our team(VEL) to the Hope Speech Detection for Equality, Diversity, and Inclusion(HSD-EDI) LT-EDI-RANLP 2023 shared task for the Bulgarian language. The main goal of this shared task is to identify the given text into the Hope speech or Non-Hope speech category. The proposed method used the H2O deep learning model with MPNet embeddings and achieved the second rank for the Bulgarian language with the Macro F1 score of 0.69.

## introduction one of the remarkable human characteristics is hope, which enables a person to envision future events and the diversity of outcomes that may be anticipated @xcite . these visions have a significant effect on a person's emotions, behaviors, and mental state, despite the fact that the desired outcome has a much-reduced likelihood of occurring. hope is essential to the well-being, recuperation, and restoration of human existence. greater optimism is consistently associated with improved academic, athletic, and physical health, psychological adjustment, and psychotherapeutic outcomes. hope theory is similar to learned optimism, optimism, self-efficacy, and self-esteem theories @xcite @xcite . people are able to freely express their opinions on numerous social networks today, which has a significant impact on human existence @xcite . the significant characteristics of social media, including rapid dissemination, low cost, accessibility, and anonymity, have increased the popularity of social media platforms such as @xcite . despite the numerous advantages of using osns, a growing body of evidence suggests that an increasing number of malicious actors are exploiting these networks to disseminate hate speech and cause harm to others @xcite . in addition, social media platforms provide a profound comprehension of people's behaviors and are important sources for natural language processing(nlp)-related scientific research @xcite @xcite . examining people's expressed levels of hope on social media is therefore believed to be an essential factor in determining their overall happiness @xcite . this type of research can shed light on the progression of goal-directed activities, resilience in the face of adversity, and the mechanisms underlying acclimation to both positive and negative life changes. in this paper, we present the work carried out on hsd-edi -lt-edi-ranlp 2023 in bulgarian language. the main goal of the shared task is to categorize the comments into hope speech or non-hope speech class. to solve this problem, our team(team-tamil) presents the approach based on an embedding technique using mpnet @xcite sentence transformer and deep learning technique with h2o @xcite deep learning model. our approach achieved the second rank with a macro f1 score of 0.69 in the bulgarian language. the rest of the paper is structured as follows: section 2 provides a brief overview of the studies related to the hope speech detection problem, section 3 provides a full explanation of the dataset, section 4 shows the proposed approach, and section 5 provides the findings of the experiments that were conducted and followed by the conclusion in section 6. two shared task was released by chakravarthi and muralidaran (2021) and chakravarthi et al. (2022b). mahajan et al. (2021) used roberta for hope speech detection in english and xlm-roberta for hope speech detection in tamil and malayalam, two low-resourced dravidian languages. their performance in classifying text into hope-speech, nonhope, and not-language is demonstrated. their methodology was rated first in english(@xmath0.93), first in tamil(@xmath1.61), and third in malayalam(@xmath2.83). junaida and ajees (2021) used deep learning-based context-aware string embeddings for word representations and recurrent neural network(rnn) and aggregated document embeddings for text representation. the authors examined and contrasted each language's three models using diverse methodologies. their approach outperforms baselines, and english, tamil, and malayalam models beat baselines by 3%, 2%, and 11%, respectively. pre-processing and transfer-learning models enable the trials. dowlagar and mamidi (2021) used pre-trained multilingual-bert with convolution neural networks for english, malayalam-english, and tamil-english code-mixed datasets, and they ranked first, third, and fourth. s et al. (2021) used transformer models, mbert for tamil and malayalam and bert for english, yielded weighted average f1scores of 0.46, 0.81, and 0.92 for tamil, malayalam, and english, respectively. vijayakumar et al. (2022 ) used bert to do this work, and their model ranked first in kannada, second in malayalam, third in tamil, and sixth in english for the hope speech 2022 shared task. b et al. (2022) used m-bert, mlnet, bert, xlmroberta, and xlm mlm to identify and classify them. bert and m-bert had the highest weighted f1-scores of 0.92, 0.71, 0.76, 0.87, and 0.83 for english, tamil, spanish, kannada, and malayalam, respectively. our study varies from the previous research in which we used mpnet and doc2vec @xcite for creating the embedding of the comments. for detecting the hope speech, we employed h2o deep learning model. ## dataset and task description we participated in the hsd-edi task in lt-edi-ranlp 2023. the main challenge of this task is to create a model that automatically detects whether the comment is a hope speech or a non-hope speech. for this task, the organizers provided the dataset with annotated labels for bulgarian, english, hindi, and spanish languages. out of these four languages, we worked only on the bulgarian language. the comments are annotated with two labels: true and false for the hope speech and non-hope speech. the in-depth details of the dataset are provided in (chakravarthi, 2020). in the first phase, a training and development set was released for creating the model. in the second phase, test sets were released only with the comments to make the prediction with the model that was created with the training set in the first phase. we need to submit the prediction that took from the test set to the organizers. in the last phase, the test set with the labels will be released test set with labels to know the performance of the model. the statistics of the datasets are shown in ## methodology we conducted an in-depth analysis of the bulgarian hope speech dataset utilizing a range of classifiers, from basic machine learning methods to powerful deep learning algorithms. the manner we carried out our research is outlined below. we utilized the scikit-learn foot_0 library for building machine learning algorithms. we used the h2o library to implement the deep neural network. we used google colaboratory foot_1 to train the models because of its user interface and quicker access to gpu resources. for detecting the hope speech from the text, firstly, we remove noise from data in order to improve data quality for improved performance and remove urls and unhelpful expressions(terms that start with @). secondly, we converted the cleaned text to feature vectors using doc2vec and mpnet. the doc2vec foot_2 is also known as paragraph vectors, an unsupervised method for learning fixed-length feature representations from texts with varying lengths, such as paragraphs, sentences, and documents. each sentence is represented by a dense vector. we set up parameters with @xmath3,vector @xmath4, @xmath5, and the remaining parameters are listed in in the next section, we explained the performance of the models. ## results and discussion in this section, we discussed the outcomes of the experiments of the model that we used. the performance of the models is evaluated with accuracy, macro precision, macro recall, macro f1, weighted precision, weighted recall, and weighted f1 score. there are 449 samples of non-hope speech and 150 samples of hope speech. we used four machine learning models and one deep learning model with two embedding techniques: doc2vec and mpnet. among all other models, h2o deep learning model performed well with both doc2vec and mpnet embeddings on the development set with macro f1 scores of 0.62 and 0.66, respectively. the results of the models with doc2vec and mpnet are shown in ## conclusion social media platforms have evolved into a forum for people to discuss their thoughts, successes, achievements, and errors. social networking members leave comments on various types of content. positive words can assist in increasing confidence and sometimes push you to be strong in difficult situations. this article describes our model that was submitted to the hsd-edi -lt-edi-ranlp 2023 competition. we used the h2o framework to build a deep neural network for categorization. we utilized mpnet from the sentence transformer for creating embeddings. our proposed method comes in second place with a weighted f1 score of 0.69 which is above the baselines. to improve performance further, the model can be fine-tuned with model architecture as well as by doing hyperparameter tuning.