FEEL - IT : Emotion and Sentiment Classification for the I talian Language (2021)

Abstract: While sentiment analysis is a popular task to understand people’s reactions online, we often need more nuanced information: is the post negative because the user is angry or sad? An abundance of approaches have been introduced for tackling these tasks, also for Italian, but they all treat only one of the tasks. We introduce FEEL-IT, a novel benchmark corpus of Italian Twitter posts annotated with four basic emotions: anger , fear , joy , sadness . By collapsing them, we can also do sentiment analysis. We evaluate our corpus on benchmark datasets for both emotion and sentiment classification, obtaining competitive results. We release an open-source Python library, so researchers can use a model trained on FEEL-IT for inferring both sentiments and emotions from Italian text.

## introduction emotions shape our lives and the way we communicate. we can be happy, sad, or angry, and we can let others know of our emotional state through language. thus, efficiently detecting emotion in text is essential for analyzing people's position towards a topic. product and service companies frequently use emotion and sentiment data to inform advertising campaigns and measure customer satisfaction @xcite . emotions have a central role in a political campaigns, and political discourse in particular @xcite . emotion and sentiment recognition can also aid in the critical decision-making process of crisis management or emergency scenarios @xcite . despite the huge interest of the natural language processing community, the majority of benchmark datasets have been proposed for english @xcite @xcite showing a limited interest for other languages, such as german @xcite , chinese @xcite , spanish @xcite , italian @xcite , and multiple languages in shared tasks @xcite .a moreover, they are usually collected either via hashtags and emojis for distant supervision @xcite @xcite , or via very specific topics @xcite @xcite . the first causes noisy training data @xcite , the second results in highly domain-specific datasets. this paper presents feel-it, a novel benchmark corpus of italian twitter posts annotated with four basic emotions @xcite : anger, fear, joy, sadness. foot_0 to the best of our knowledge, no other italian dataset with a broad topic and domain coverage for emotion and sentiment classification exists. beyond releasing benchmark results on feel-it, we evaluate recent neural models trained on our corpus for emotion recognition example emotion pagliacci ammaestrati dal grillo parlante di pinocchio anger they are buffoons controlled by pinocchio's jiminy cricket non ci sto dormendo la notte. #22agosto #covid19 fear this does not make me sleep at night. #22august #covid19 adoro questa canzone, è una delle mie preferite stream icarus falls joy i love this song, it's one of my favourite stream icarus falls i brividi. come si può spegnere una vita con così tanta facilità? non ho parole.... sadness i got chills. how can you kill someone so easily? i do not know what to say.... on the multiemotions-it dataset @xcite . it contains comments on music videos and advertisements posted on youtube and facebook. we also test performance on sentiment classification by collapsing positive and negative emotions on the sentipolc16 benchmark dataset @xcite . it comprises both general and political topics. the best-performing models are released as part of a python library to foster and facilitate research on the topic. contributions. we present feel-it, a new corpus on italian tweets, annotated with four basic emotions (anger, fear, joy, sadness). we demonstrate that we can effectively predict sentiments and emotions in text by training prediction models on this corpus. we release an open-source python library foot_1 that researchers can use to classify their text. ## data collection and annotation we retrieved the data by monitoring trending topics each day between 20 th august to 12 th october 2020, using the twitter api. for each day, we sampled 1000 tweets. this approach allowed us to get data from a range of different topics that span over many weeks. the two first authors labeled the complete set of posts. both are native italian speakers with a strong nlp background. eventually, the number of annotated tweets that contained an emotion was 2037 tweets (we removed tweets that did not contain any emotion, that is, most of them). this process involves a lot of data that has been discarded, and it is time-consuming, but the upside of it is that the collected tweets are from diverse domains and are high quality. we computed our inter-rater agreement on a shared set of 220 tweets, annotated both with emotions and with none (i.e., no emotion found). we reached an agreement of 0.6 (krippendorff's alpha). once none tweets were removed, the agreement on the remaining 68 annotated tweets was 0.8 (krippendorff's alpha). corpus analysis similar to other realistic emotion classification datasets @xcite @xcite , the dataset is imbalanced. the distribution is similar to the semeval-2018 task 1 dataset @xcite , where anger and joy account for the majority of tweets, and fear is the least frequent emotion. 3 in feel-it, topics vary both with respect to domains and time. topic domains ranges from health (#covid19, #mascherina/mask) to sports (#f1, #juventus), from social issues (#scuola/school) to tv shows (#gfvip, #pomeriggio5), from individuals (#dimaio, #suarez) to generic targets (#negazionisti/negationists). each topic is associated with a time range that greatly varies with subject. tv shows are cited when they are broadcast, e.g., #domenicalive, literally sunday live is mainly commented on sunday. some events, like soccer matches or celebrity birthdays, are mentioned only one day, e.g., the hashtag of the soccer match #beneventointer appears 371 times, but only the 31 st september. tweets related to covid-19 are present every observed day, with some peaks for specific events (e.g., on the 2 nd october, we recorded a peak of 132 tweets due to the news of us president trump testing positive for covid-19). ## experiments we use experimental evaluation to (i) show that our classifier can predict emotions in tweets and (ii) that feel-it can also be used to perform sentiment classification with competitive results. ## use-cases: covid-19 and multiemotions-it to further validate our approach, we showcase the results of emotion recognition models trained on feel-it and tested on two topic-specific datasets: multiemotions-it @xcite and a dataset of 662 tweets about covid-19. multiemotions-it (me) @xcite is a linguistic resource for italian which comprises comments of music videos and advertisements posted on youtube and facebook. each text is manually annotated according to four different dimensions: i.e., relatedness, opinion polarity, emotions, and sarcasm. this dataset differs from feel-it both in terms of topic variety and considered social media. among all the emotion classes considered in me, we removed the ones not pertaining to our set of emotions. after this process, we are left with 304 comments. as before, we pick umberto-ft and umberto-pt as our champion models. to give a point of reference, we also show the most frequent class (mfc) baseline results. results. ## related work different works have explored emotion recognition approaches. however, few of them incorporate text in italian. indeed, currently, no generalpurpose dataset for emotion recognition has been proposed for the italian language. however, for italian, there is a dataset for emotion recognition limited to youtube and facebook comments (sprugnoli, 2020), and one for sentiment analysis sen-tipolc16 @xcite . we used these datasets in the experimental evaluation to show that our model can also perform sentiment prediction. regarding other languages, abdul-mageed and ungar (2017) proposes emonet, an english emotion dataset that has been collected using a keyword-based approach (e.g., tweets are retrieved using #happy as a marker for joy). the authors have obtained high accuracy with this dataset. alternatevely, we approach the problem annotating manually and without using distant supervision. emotxt @xcite is an open-source toolkit for emotion prediction supporting prediction for different emotions for the english language: love, joy, surprise, anger, sadness, and fear. nozza et al. (2017) ## conclusions we present feel-it, a new corpus for emotion classification on italian twitter data, and release an open-source python library to run both emotion and sentiment classification. future work will focus on the extension of this dataset, considering other emotions and languages.