RED : A Novel Dataset for R omanian Emotion Detection from Tweets (2021)

Abstract: In Romanian language there are some resources for automatic text comprehension, but for Emotion Detection, not lexicon-based, there are none. To cover this gap, we extracted data from Twitter and created the first dataset containing tweets annotated with five types of emotions: joy, fear, sadness, anger and neutral, with the intent of being used for opinion mining and analysis tasks. In this article we present some features of our novel dataset, and create a benchmark to achieve the first supervised machine learning model for automatic Emotion Detection in Romanian short texts. We investigate the performance of four classical machine learning models: Multinomial Naive Bayes, Logistic Regression, Support Vector Classification and Linear Support Vector Classification. We also investigate more modern approaches like fastText, which makes use of subword information. Lastly, we fine-tune the Romanian BERT for text classification and our experiments show that the BERT-based model has the best performance for the task of Emotion Detection from Romanian tweets. Keywords: Emotion Detection, Twitter, Romanian, Supervised Machine Learning

## introduction romanian language is a very little explored language in terms of natural language processing (nlp) and natural language understanding (nlu), but social media is full of data and people sharing their opinions about diverse topics, thus some opinion mining tools would be very useful. opinion mining is defined in @xcite as "the task of detecting, extracting and classifying opinions, sentiments and attitudes concerning different topics, as expressed in textual input". in nlp tasks, there are two major subcategories of opinion mining: sentiment analysis (sa) and emotion detec-tion (ed). the main difference between these two is that sentiment analysis categorizes opinions between positive or negative, while emotion detection aims to extract the specific emotion a text gives to the reader. english language is rich in nlp resources -we can see not only that there are many manually labelled datasets, with a large variety of emotions, but there also exist datasets with tweets labeled by emotion intensity (for example, wassa dataset by @xcite which was first presented at the 2017 shared task on emotion intensity). detecting emotions can be a subjective task, even psychologists didn't agree upon what the main emotions of a human being are. @xcite identified eight primary emotions: ecstasy, admiration, terror, amazement, grief, loathing, rage and vigilance, while @xcite said that the primary emotions are: joy, sadness, fear, disgust, surprise and anger. inspired by the work of @xcite , we decided to collect tweets from twitter and manually annotate them for four emotions: fear, anger, joy, sadness, and also add a neutral class. we show the quality of the overall dataset by computing statistic descriptors and finally we train machine learning classifiers in order to create the first qualitative romanian tool for emotion detection. the usefulness of having a tool that detects emotions from texts varies from exploring customer opinions in order to ensure business growth, to improving human-computer interactions (hci) in applications like onboarding chatbots and personal digital assistants @xcite ). ## recent works it has been shown by @xcite 2020 ) that text based ed research studies have been given less attention than other methods of ed, for instance multimodal emotion detection including speech, body language, facial expressions, and so on. there are two principal ways of assessing the ed problem: by using rule construction techniques to identify emotive words and word combinations, and machine learning approaches, which can be supervised or unsupervised. while unsupervised methods learn from unstructured data and could provide important clustering insights in opinion mining tasks, supervised methods give better detection rates @xcite ), but only if there exist some quality labeled data to give as input to the classifier. combining these two main approaches gives a third way of detecting emotions from text -the hybrid approach, which takes the advantages of rule-construction and amplifies them with the power of machine learning. the latter was used in this paper. @xcite used the isear dataset to train a supervised model of emotion detection with logistic regression. the isear (international survey on emotion antecedents and reactions) dataset was first introduced in the work of @xcite and contains 7666 emotional sentences labelled with 7 types of emotions: anger, disgust, fear, sadness, shame, joy and guilt. @xcite used the same dataset to test their dominant meaning approach in detecting emotions in chat messages. for detecting emotions in tweets, @xcite proposed a hybrid approach consisting of lexical based approaches that use wordnet-affect and emosenticnet with supervised classifiers trained on ait-2018 dataset for english. this dataset was introduced in semeval-2018 task 1: affect in tweets by @xcite and consists of tweets annotated in 3 languages, for anger, fear, joy and sadness. @xcite presented an ensemble classifier based on nlp techniques like doc2vec and 1500 k-nearest neighbor, multilayer perceptron and decision tree basic classifiers optimized with parzen tree estimator (tpe), to detect emotions from isear and oanc datasets, and also from an unstructured dataset of tweets from crowdflower. their results show an outstanding accuracy of 99.49% for regular sentences and 88.49% for irregular sentences. @xcite tried emotion detection in text using word-embeddings like word2vec, glove and fasttext on their designed model, having as training datasets isear, ait-2018 and semeval-2019 task3 dataset. their research concluded that fasttext had slightly better performances. huang et al. (2019) investigated fine-tuning bert (bidirectional encoder representations from transformers) on two datasets: emotionlines with dialogues from friends sitcom, and emotion-push containing facebook messenger chats. their model obtained a micro f-score of 0.815 on friends and 0.885 on emotionpush. acheampong et al. (2020) fine-tuned bert with a bi-lstm classifier and obtain an average f-score of 0.73 on the isear dataset. chiorrini et al. (2021) also investigated the use of bert for both sentiment analysis and emotion detection in twitter data, using wassa dataset @xcite ), a shorter version of ait-2018 containing 6755 tweets annotated for: sadness, fear, anger and happiness. for more exotic languages, however, researchers had to manually annotate sentences or words in order to have a high quality dataset to work on. for arabic, @xcite developed a dataset of 4000 emotional words, including emojis, which they used to classify real-time tweets into 5 types of emotions (happy, sad, angry, scared, surprised). @xcite used a hybrid approach to detect emotions from punjabi texts -first they consider a rule-based engine to detect if the sentence has an emotion or not, and then they applied support vector machine (svm) and naïve bayes (nb) classifiers to detect 6 emotions: happy, fear, anger, sadness, disgust and surprise. @xcite created a corpus of manually labelled malayalam texts (an indian dialect) into emotions like: sad, happy, anger or fear. further they used a svm classifier to create emotion detection models. for romanian language, there were some studies conducted on speech datasets with annotated emotions. for instance, @xcite presented a method for emotion detection in romanian speech that use largest lyapunov exponent of the mel-frequency energy bands, with svm and wknn (weighted k-nearest neighbors) classifiers, trained on the srol dataset (developed by @xcite ) @xcite created a deep learning model of convolutional neural networks (cnn) trained on a set of recordings in romanian language. @xcite used three sets of recordings in romanian language, annotated for positive and negative emotions, and trained models using k-nn and svm classifiers. in terms of text data, @xcite present an unsupervised clustering approach used to mine emotional patterns in mihai eminescu's poetry, based on the romanian emotion lexicon created by @xcite for feature extraction. in terms of sentiment analysis for romanian language, istrati and ciobotaru (2021) created a dataset of tweets annotated for positive/negative sentiment and trained several classifiers on it, both classical and modern. there also exist some lexicon based approaches for romanian sentiment analysis, like babelsentic-net, a multilingual concept-level knowledge base described in @xcite . inspired by some of the works presented in this section, we created the first dataset of romanian tweets labelled with five different emotions (anger, fear, sadness, hapiness and neutral), in order to obtain a solid tool for detecting emotions in texts. ## data in this section we present in detail our novel dataset, red (romanian emotion detection). ## models in this section we describe the tried models in order to generate the first supervised emotion detector for romanian short texts. first, we create a benchmark composed of classical machine learning models, and second we try two more modern approaches, which make use of word embeddings: one that uses facebook's fasttext word embeddings and a fine-tuned bert classifier. ## experiments and results in this section we present the results obtained on the test set for the previously described models. ## conclusions and future works in this article, we presented our novel dataset for emotion detection, the first dataset of its kind for the romanian language. we researched the stateof-the-art and created a benchmark of machine learning models in order to obtain an automatic emotion detector from tweets, having the purpose of being used in real-life tasks, adjacent to the field of opinion mining. although the dataset is not very large, it provided enough data to generate a text emotion detection model with an accuracy of 90.37%. in the future, we plan to enlarge this dataset with more tweets per class, an action that most probably will increase accuracy of the models. also, we plan on adding more classes of emotions, to generate an even more fine-grained dataset for detecting emotions in romanian content.