Nearest neighbour approaches for Emotion Detection in Tweets (2021)

Abstract: Emotion detection is an important task that can be applied to social media data to discover new knowledge. While the use of deep learning methods for this task has been prevalent, they are black-box models, making their decisions hard to interpret for a human operator. Therefore, in this paper, we propose an approach using weighted k Nearest Neighbours (kNN), a simple, easy to implement, and explainable machine learning model. These qualities can help to enhance results’ reliability and guide error analysis. In particular, we apply the weighted kNN model to the shared emotion detection task in tweets from SemEval-2018. Tweets are represented using different text embedding methods and emotion lexicon vocabulary scores, and classification is done by an ensemble of weighted kNN models. Our best approaches obtain results competitive with state-of-the-art solutions and open up a promising alternative path to neural network methods.

## introduction in this paper, we consider semeval-2018 task 1 ei-oc: affect in tweets for english 1 @xcite . this is a classification problem in which data instances are raw tweets, labeled with scores expressing how much each of four considered emotions (anger, sadness, joy, and fear) are present. our target is to implement the weighted k nearest neighbor (wknn) algorithm to detect emotions in tweets. in doing so, we consider different ways of tweet embeddings and combine them with various emotional lexicons, which provide an emotional score for each word. the motivation for using wknn is to show the potential of a simple, interpretable machine learning approach compared to black-box techniques based on more complex models like neural networks @xcite . by contrast to the latter, wknn's predictions for a test sample can be traced back easily to the training samples (the nearest neighbours) that triggered this decision. we note that we still use nn-based methods for obtaining tweet embeddings. one could therefore argue that our method is not fully explainable; however, we feel that it is less important to understand how tweets are initially represented in an n-dimensional space, than to explain how they are used in making predictions for nearby instances. the remainder of this paper is organized as follows: in section 2 we discuss related work, mainly focusing on the winning approaches of semeval-2018 task 1. in section 3, we describe the methodology behind our solution, including data cleaning, tweet representations through word embeddings, lexicon vocabularies, and their combinations; our proposed ensemble method for classification; and finally, evaluation measures. in section 4, we report the observed performance on training and development data for the different setups of our proposal, while section 5 lists the results of the best approach on the test data and compares them to the competition results. in section 6, we examine some of the test samples with correct and wrong predictions to see how we can use our model's interpretability to explain the obtained results. finally, in section 7, we discuss our results and consider possible ways to improve them. ## related work first, we briefly recall the most successful proposals foot_1 to the semeval-2018 task. the winning approach @xcite uses tweet embedding vectors in ensembles of xgboost and random forest classification models. the runners-up @xcite perform transfer learning with long short term memory (lstm) neural networks. the third-place contestants @xcite ) train an ensemble of a complex model consisting of gated-recurrent-units (gru) using a convolution neural network (cnn) as an attention mechanism. it is clear that the leaderboard is dominated by solutions that are neither simple nor interpretable. this comes as no surprise, given that the effectiveness of a solution is evaluated only using the pearson correlation coefficient (see formula (3) in section 3.5). in general, machine learning models in the natural language processing (nlp) field rarely explain their predicted labels. this inspires the need for explainable models, which concentrate on interpreting outputs and the connection of inputs with outputs. for example, @xcite present an explainable classification approach that solves nlp tasks with comparable accuracy to neural networks and also generates explanations for its solutions. recently, @xcite presented an overview of explainable methods for nlp tasks. apart from focusing on explanations of model predictions, they also discuss the most important techniques to generate and visualize explanations. the paper also discusses evaluation techniques to measure the quality of the obtained explanations, which could be useful in future work. in this paper, we consider one of the simplest explainable models: the knn method. in the context of nlp, knn has recently been applied by @xcite as a backoff method for classifiers based on bert and roberta (see section 3.2). in particular, when the latter nn methods are less confident about their predictions, the knn solution is used instead. in this paper, we will only use such nn approaches at the data representation level and rely on weighted knn only during classification. ## methodology in this section, we describe the different ingredients of our approach, more precisely, data preprocessing, embedding methods, emotional lexicon vocabularies, classification, and evaluation methods. we focus on the emotion intensity ordinal classification task (ei-oc) @xcite . given each of the four considered emotions (anger, fear, joy, sadness), the task is to classify a tweet in english into one of four ordinal classes of emotion intensity (0: no emotion can be inferred, 1: low amount of emotion can be inferred, 2: moderate amount of emotion can be inferred, 3: high amount of emotion can be inferred) which best represents the mental state of the tweeter. separate training, development, and test datasets were provided for each emotion. to train the classification model, we merge the training and development datasets to evaluate our results with the cross-validation method. ## experiments our experiments on the train and development are designed as follows: first, we compare the individual tweet embedding methods (section 4.1) and examine which setup gives the best results. then, in section 4.2, we also involve the emotional lexicons, either independently, by appending them to tweet embedding vectors and in ensembles. ## results on the test data to determine the generalization strength of the obtained best approach from section 4, we evaluated it on the test data by submitting predicted labels in the required format to the competition page foot_10 . pcc scores were calculated for each emotion dataset, and obtained results were averaged. because of the mean voting function in our model's ensemble, our predicted labels are in float format. therefore, to match the requested format on the competition page, before submitting results, we rounded them to the nearest integer label. the obtained pcc scores are shown in ## discussion and error analysis to illustrate our approach's explainability, in this section we explore some correctly and wrongly predicted test samples. as an example of a correct prediction, we can take a look at an anger test tweet: "i know you mean well, but i'm offended. prick." with real anger class "2". our best model predicted label 2.4, which was rounded to 2, so our result is correct. to analyze how this label is obtained, we look at the predictions by all models separately. they are shown in next, we also examined the neighbours chosen by the models and their classes, especially those which are selected by different models. to find some patterns, we took the intersection of the neighbours closest to the test instance, chosen by the ensemble's models. we should mention that those models are based on different embeddings, which may locate tweets in n-dimensional space differently. however, one tweet with class "2" from the train data was chosen by 4 models out of 7 and five more tweets (four of them with class "2" and one with class "1") by 3 models. a closer examination of those tweets revealed that all of them contain the word "offended". from this, we could conclude that this word has a high emotional intensity that influences the sentence's tone. the next sample we examined is another anger test tweet, with gold label "0": "we've been broken up a while, both moved on, she's got a kid, i don't hold any animosity towards her anymore..." our solution predicted a score 1.5, which was rounded to 2, leading to a false prediction. similar to the previous sample, we took a look at the classes predicted by the different models in the ensemble ( we also explored the most frequent neighbours, which were chosen by 3 models (one tweet with class "1") and by 2 models (nine tweets with different classes). we did not find any noticeable patterns; the misclassification is probably caused by words with high emotional intensity, like "animosity", which is used in combination with a negation in this specific context. ## conclusion and future work in this paper, we evaluate an explainable machine learning method application for the emotion detection task. as the main conclusion, we can say that using simple optimizations and the weighted knn method can perform nearly on par with more complex state-of-the-art neural network-based approaches. in the future, we plan to incorporate more elaborate nearest neighbour methodologies, which also take into account the inherently fuzzy nature of emotion data. some initial experiments with ordered weighted average based fuzzy rough sets @xcite show promising results. another observation that can be made from our results is that the most informative input to solving the emotion detection task is provided by the tweet embeddings, and that lexicons generally do not improve the results a lot. meanwhile, adding the combined vector of roberta embedding and the best lexicon scores increased pcc scores noticeably. as a possible further improvement, we may refine the voting function by assigning different weights to the different members of the ensemble, which can be based, for example, on the confidence scores. furthermore, as another strategy to improve results, additional text preprocessing steps could be performed, for example, using exclamation marks or word lemmatization. also, we can give more weight to the hashtag and emoji descriptions during the tweet embedding process. another important characteristic that influences the results is data imbalance. as observed, we obtained the lowest pcc scores on the fear dataset, most likely because it is the most imbalanced one. for further experiments with fear, we consider the usage of imbalanced machine learning classification methods. in particular, @xcite discusses several approaches based on fuzzy rough set theory. finally, @xcite provide several hints to investigate and improve solution explainability. for example, we can examine feature importance, measure the quality of explainability, etc.