XED : A Multilingual Dataset for Sentiment Analysis and Emotion Detection (2020)

Abstract: We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of human-annotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing new resources for many low-resource languages. We use Plutchik’s core emotions to annotate the dataset with the addition of neutral to create a multilabel multiclass dataset. The dataset is carefully evaluated using language-specific BERT models and SVMs to show that XED performs on par with other similar datasets and is therefore a useful tool for sentiment analysis and emotion detection.

## introduction there is an ever increasing need for labeled datasets for machine learning. this is true for english as well as other, often under-resourced, languages. we provide a cross-lingual fine-grained sentence-level emotion and sentiment dataset. the dataset consists of parallel manually annotated data for english and finnish, with additional parallel datasets of varying sizes for a total of 32 languages created by annotation projection. we use plutchik's wheel of emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) @xcite as our annotation scheme with the addition of neutral on movie subtitle data from opus @xcite . we perform evaluations with fine-tuned cased multilingual and language specific bert (bidirectional encoder representations from transformers) models @xcite , as well as suport vector machines (svms). our evaluations show that the human-annotated datasets behave on par with comparable state-of-the-art datasets such as the goemotions dataset @xcite . furthermore, the projected datasets have accuracies that closely resemble human-annotated data with macro f1 scores of 0.51 for the human annotated finnish data and 0.45 for the projected finnish data when evaluating with finbert @xcite . the xed dataset can be used in emotion classification tasks and other applications that can benefit from sentiment analysis and emotion detection such as offensive language identification. the data is open source foot_0 licensed under a creative commons attribution 4.0 international license (cc-by). in the following sections we discuss related work and describe our datasets. the datasets are then evaluated and the results discussed in the discussion section. ## background & previous work datasets created for sentiment analysis have been available for researchers since at least the early 2000s @xcite . such datasets generally use a binary or ternary annotation scheme (positive, negative + neutral) (e.g. @xcite ) and have traditionally been based on review data such as, e.g. amazon product reviews, or movie reviews @xcite @xcite . many, if not most, emotion datasets on the other hand use twitter as a source and individual tweets as level of granularity @xcite @xcite . in the case of emotion datasets, the emotion taxonomies used are often based on @xcite and @xcite (which is partially based on ekman). ## the data in we used plutchik's core emotions as our annotation scheme resulting in 8 distinct emotion categories plus neutral. the sentimentator platform @xcite allows for the annotation of intensities resulting in what is essentially 30 emotions and sentiments, however, as the intensity score is not available for all annotations, the intensity scores were discarded. the granularity of our annotations roughly correspond to sentence-level annotations, although as our source data is movie subtitles, our shortest subtitle is ! and the longest subtitle consists of three separate sentences. a majority of the subtitles for english were assigned one emotion label (78%), 17% were assigned two, and roughly 5% had three or more categories (see also ## evaluation a dataset for classification tasks is useful only if the accuracy of its annotations can be confirmed. to this end we use bert to evaluate our annotations as it has consistently outperformed other models in recent classification tasks (see e.g @xcite ), and support vector machines for its simplicity and effectiveness. we use a stratified split of 70:20:10 for training, dev, and test data. we use a fine-tuned english uncased bert, with a batch size of 96. the learning rate of adam optimizer was set to 2e-5 and the model was trained for 3 epochs. the sequence length was set to 48. we perform a 5-fold cross validation. we also use an svm classifier with linear kernel and regularization parameter of 1. word unigrams, bigrams and trigrams were used as features in this case. implementation was done using the linearsvc class from the scikit-learn library @xcite . binary refers to positive and negative, and ternary refers to positive, negativeand neutral. for binary evaluations we categorized anger, disgust, fear, and sadness as negative, and anticipation, joy, and trust as positive. surprise was either discarded or included as a separate category (see ## discussion the results from the dataset evaluations show that the xed is on par with other similar datasets, but they also stress that reliable emotion detection is still a very challenging task. it is not necessarily an issue with natural language processing and understanding as these types of tasks are challenging for human annotators alike. if human annotators cannot agree on labels, it is not reasonable to think computers can do any better regardless of annotation scheme or model used since these models are restricted by human performance. the best accuracies are those that are in line with annotator agreement. xed is a novel state-of-the-art dataset that provides a new challenge in fine-grained emotion detection with previously unavailable language coverage. what makes the xed dataset particularly valuable is the large number of annotations at high granularity, as most other similar datasets are annotated at a much coarser granularity. the use of movie subtitles as source data means that it is possible to use the xed dataset across multiple domains (e.g. social media) as the source data is representative of other domains and not as restricted to the domain of the source data (movies) as many other datasets. perhaps the greatest contribution of all is that, for the first time, many under-resourced languages have emotion datasets that can be used in other possible downstream applications as well.