Frustration Level Annotation in L atvian Tweets with Non-Lexical Means of Expression (2021)

Abstract: We present a neural-network-driven model for annotating frustration intensity in customer support tweets, based on representing tweet texts using a bag-of-words encoding after processing with subword segmentation together with non-lexical features. The model was evaluated on tweets in English and Latvian languages, focusing on aspects beyond the pure bag-of-words representations used in previous research. The experimental results show that the model can be successfully applied for texts in a non-English language, and that adding non-lexical features to tweet representations significantly improves performance, while subword segmentation has a moderate but positive effect on model accuracy. Our code and training data are publicly available.

## introduction dramatically increasing data storage and processing capacities have resulted in an explosion of available data, and of potential uses for the many kinds of knowledge or insights that could theoretically be extracted from that data. the development of web 2.0 has created unprecedented amounts of text, and, in recent decades, images and videos. but the sheer volume of the available data is problematic due to a shortage of human resources (time and attention) available for analyzing or even just browsing through it all, as described in @xcite . from the very start, as soon as machine learning techniques appeared, they were immediately applied to text analysis. neural networks have proven particularly suitable for such tasks, but to learn to approximate human judgements, these 1 source code is available at https://github.com/lynx1981/dfrustration/ methods generally need a large amount of manually annotated training data to obtain a viable model. creating such corpora is a very timeconsuming task, especially if it has to be done from scratch. for languages with a relatively low number of speakers, such as latvian, this is a particularly pressing problem, because of the small number of textual corpora available, so that researchers generally need to create their own datasets. much of the information of importance to be captured from the internet is by its nature related to emotions. recommendation systems, conversational "chat bots", automated customer support assistants, various tools for analyzing, monitoring or enhancing personal well-being or mental health, not to mention systems for targeting or crafting advertising or marketing messagesall can benefit from understanding the emotional state of the recipient or sender of a given message. but recognition of emotions in text generally requires considerable investment of human resources; and is difficult to automate in principle. perception of emotions is largely subjective, and the emotions perceived from the same source by different people differ. further, in written communication, and especially communication in social networks, the words alone do not capture all of the emotional information available in the message. for example, the emotional tone of a sentence may vary depending on whether it was followed by period as the absence of periods is a characteristic feature of internet jargon usage, as shown in @xcite , which may carry a different emotional charge. in this work, we propose the method for automatic annotation of the emotional charge found in text with the help of non-lexical means of expression -typographical marks, emojis and likewise. in addition, to be able to do so for latvian, we propose an annotated dataset that was used for model training and its results. ## background and related works as machine learning methods, and neural networks, in particular, have developed over the past couple of decades, they naturally started to be applied to recognizing emotions, especially following the demonstrated successes of neural networks in image recognition as in @xcite . emotion recognition in speech, as the easier task, began to appear as early as 2000 in @xcite . we begin to see published research on the recognition of emotions in text only beginning with @xcite . the majority of works focus on the classification of text according to its predominant emotion, which means that the text or part of the text was classified as containing one of the basic emotions. the most commonly used is ekman's emotion classification scheme, described in (ekman, 1992), which contains six basic emotions: anger, fear, sadness, disgust, surprise, and joy -either in the standard way, or with various extensions, such as in @xcite , or reductions, as in @xcite . another model used to classify emotions, in two variants, is the russell circumplex model, described in @xcite , which is a two-or three-factor model where each emotion is represented in a two-dimensional (valencearousal) or three-dimensional space, with axes of dominance, valence and arousal. although there are numerous works that use these emotion classification models -for example, the threefactor model was used in (parthasaraty and busso, 2017) and the two-factor model in @xcite -these are not as common as works using a categorical list of basic emotions. as computing power and data grew, the classification of emotions began to shift from a purely qualitative to a more quantitative approach: the intensity of emotions, not just the presence of specific emotions in the text, began to be studied. some authors even create automatic tools for classifying emotions with intensity, for example the weka package presented in @xcite for four emotions (anger, fear, joy and sadness), or another opensource emotion computing framework presented in @xcite for the same four emotions. interestingly, however, none of these classifications include frustration, or dissatisfaction, in the list of emotions, even if the emotion itself is well known to everyone and plays an important role, for example, in assessing quality of service in @xcite . in customer service, however, 'dissatisfaction' is typically used instead of the word 'frustration', but this does not change the content of the term. there are very few works on annotating frustration, and they are relatively old, for example, @xcite and @xcite , where the authors discussed the possibility of reducing human frustration in dialogue with the help of an emotional or empathic agent. in @xcite , the authors achieved good results in identifying frustration, but they used a complex multimodal system that measured the physical parameters of human (in their case, student) behavior, such as the pressure on the chair and the speed of the mouse. in short, the recognition of frustration from text has not been adequately studied. a recent paper @xcite ) is a relatively rare example in which intensities for eight differing emotional "tones" (anxious, frustrated, impolite, passionate, polite, sad, satisfied, and empathetic) were annotated; however, the goals and methods of this work and ours were significantly different: while we examine the effect of words and non-lexical means of expression on perceived frustration, they look for correlations between user and support worker emotional tones; also, their approach uses a seq2seq ("sequence to sequence") neural model, while we employ a model based on an architecturally simpler, fully-connected feedforward neural classification network. the works we have mentioned have one feature in common: the classification of emotions in them is based on the analysis of words, or lexical means of expression. several studies have been devoted to compiling lexicons, such as @xcite and @xcite ), but we have not been able to find a lexicon of non-lexical features. @xcite mentions the use of non-lexical means of expression for the classification of emotions (using eckman's scheme) but does not give a list or description of these means. with the growing popularity of twitter as a source of data, several works are making use of emojis, like @xcite or hashtags, like @xcite . the article @xcite is devoted to the construction of a hashtag lexicon for the automatic classification of emotions, however, there is one problem with their use: most twitter messages do not contain any hashtags, at least in the domain of customer support conversations. textual features such as the use of exclamation and question marks were used in @xcite , and message length in @xcite . the use of nonlexical (and non-linguistic) means of expression for the classification and intensity of emotions is much more developed for voice communication. an example is @xcite , where both nonlexical features such as speaking speed or number and length of pauses, and non-linguistic features such as inhalation and exhalation are used to classify emotions into eight classes. attempts to use other linguistic features, such as the ratio of sentence parts to each other, like in @xcite or the number of word separators or word separator sequences as in @xcite in addition to a basic bag-of-words representation, are being made, but a systematic review and study of such features is still lacking. ## preceding work we use two gauge points for appraising the performance of our model: first is the baseline accuracy, obtained by always predicting the most frequent ground-truth rating, and second -the results obtained by an equivalent model based solely on lexical features and not employing any sort of input processing. a paper by @xcite ) presented such a model for predicting frustration intensity level based on lexical features only. there, the authors employed a fully connected feedforward neural network with 64 hidden units. this network took as input a bagof-words representation of the input text, using a subset vocabulary constructed during the training phase. in order to construct this, for every word in the dataset that was encountered in more than 2 entries, the following statistics were calculated: the average value of frustration intensity of the entries this word was found in, and the standard deviation of this value. entries that were annotated as not rated ("n") or missing a rating value were ignored. the standard deviation is a main criterion for constructing the bag-of-words "best words" vocabulary, based on reasoning that the lower the standard deviation of the frustration rating, the more characteristic the specific word is for the given frustration intensity. for each set of training data, a vocabulary was constructed, and then used for preprocessing each input entry for calculating predictions. the output of the model was a value from 0 to 4, representing the predicted frustration intensity. the performance of this model is used as a baseline for comparison. ## model it can be seen that a user message in the model is used to construct two types of features: lexiconbased and non-lexical means of expression-based. we use the same method of lexicon construction as developed for the baseline model which was described in the previous work section. however, the new model differs in that it applies subword segmentation of the user message prior to constructing the bag-of-words representation (which thus becomes a 'bag-of-subword-units'), and it also adds features to the model input based on a range of non-lexical means of expression. we next discuss these in detail. ## dataset in this work, we have used a completely new latvian dataset, developed specifically for the purpose of testing the performance of our proposed frustration annotation model against the old one used as a gauge point. following the example of many other recent researchers, we selected twitter as a data source of choice. four major latvian internet and telecommunication service provider accounts were chosen for collecting conversations of users with customer support representatives. those accounts are: (@mans_tet), (@mans_lmt), (@bitelv) and (@tele2latvija), which belong to companies tet, lmt, bite and tele2, respectively. to provide for not just the possibility of appraising a frustration level at a certain moment of time, but also to allow studying the dynamics of frustration changes from one user turn to the next, the selected conversations contain no less than two user turns, with at least one customer support turn between those. as a turn we consider a sequence of messages that belong to one party in the conversation (not interrupted by any other users' messages). an essential criterion was that the dialogs should be in latvian, as obtaining a dataset in latvian was the primary goal of collecting this dataset of tweets. the conversations were collected manually, in order to ensure that the criteria are met and that eligible conversations were not excluded just because another user replied to a tweet from the conversation, if such intervention did not actually affect the initial dialog. the resulting dataset consists of 283 dialogs with 688 user turns and 531 customer support representative turns. of those 688 user turns, 9% had a median frustration value of 0, 19% of 1, 31% of 2, 30% of 3 and, finally, 11% of 4. the resulting collection was post-processed and saved in a unified and anonymized format, as described in the experimental setup section. each user's turn in the dataset is followed by three values, representing the level of frustration assigned to this turn by three independent annotators. each value represents a frustration level measured on a scale of 0 to 4, or can be "n" if the annotator judged that a level of frustration could not be determined from the text of the user message, for example, in case of the user simply stating their address. a final option is a missing value, for example if the text was in a language other than latvian or could not be understood by an annotator for some other reason. for english, in order for the results to be comparable with the baseline, we have employed the same dataset that was used in @xcite . the dataset represents a small subset of the kaggle customer support dataset, where approximately 400 consistent dialogues between a support and a user were isolated and annotated for frustration by three independent annotators on the scale of 0..4. in total, this dataset contained 843 user turns, of which 18% had median frustration value of 0, 15% of 1, 28% of 2, 27% of 3 and 11% of 4. thus, the baseline accuracy of the most frequent value was 31% for latvian and 28% for english, and the most frequent frustration values were 2 and 3, respectively. ## experimental setup the model was implemented on python as a neural network with relu activation function and a softmax layer. it is using the following metaparameters: number of hidden units, number of epochs. the model, using both non-lexical and lexical means of expression, whether it was using subword segmentation of the input text or not, was trained on the same set of data. we tested the model on different vocabulary sizes, fixing the number of epochs as 100 and the number of hidden units as 64, and found that the tendency of vocabulary sizes 50 and 500 to underperform has been preserved. we further explored the effect of hyperparameter changes, and also some additional input processing methods. we tried different network sizes (number of hidden units in the dense feedforward layer), including 32, 128, and 256, and concluded that 32 hidden units were not sufficient, while 128 and 256 gave suboptimal results with 0.75%, 0.5% and 1.5% decline in accuracy, respectively. the model takes as its input a file with a collection of dialogs between users and customer support representatives in an anonymized format. information such as twitter user ids and message ids were removed from the dialogs, and any included sensitive information such as e-mail or customer number, is replaced with generic placeholders, following the example of the kaggle customer support dataset. all user messages in the dataset are tagged with either "user:" or "supp:" to denote whether they belong to a client/user or to the company's customer support representative, respectively. consecutive messages coming from a single party are joined together, forming a single "turn" -a sequence of messages uninterrupted by another party, so that the model works under the assumption that each two consecutive messages in a dialog belong to the different parties. for all experiments reported here, we use a single value per turn, calculated as a median of three ratings given by the annotators, which allows keeping the aggregated annotation value as an integer, thus enabling us to perform classification rather than regression. the number of occurrences of each of the median ratings for values 0 to 4 is @xcite @xcite @xcite , respectively, with 2 being the most frequent rating with 210 instances. to determine the optimal selection of features, used in the model, we have used a two-step process. to appraise the performance of the nlme features, for comparison we used the model with bag-of-words input features only. figure 2 provides its principal schema of operation. in the first step, we identified all potential features in the dataset and calculated a correlation table with the median annotated value, leaving only those that had at least a weak correlation. in the second step, we have run the model with a different combination of those features. first, we have used none and all the features for a benchmark, and then tested every feature in isolation to verify that no single feature would give a comparable accuracy. since none did, we next tried to exclude features that in isolation gave worse results than a model using bag-of-words only. however, the removal of underperforming features (those which in isolation give results below the performance of the bag-of-words-only model) resulted in decreasing the overall performance by 0.4% (z-@xmath0.32). for segmentation of latvian text, we have applied a genseg tool, described in @xcite to preprocess the dialog file, so that the input now consisted of the messages in an already segmented form, leaving the rest of the process exactly as before -so that the run of the model on segmented versus unsegmented data differed only in the input file. having fixed the metaparameters at 64 hidden units and vocabulary size 100, we found that subword segmentation improved the resulting model accuracy by 1.25% (z-@xmath1.02). ## results as our goal was to test improvements on latvian data, and additionally adapt the original model to the specific challenges of the latvian language, we created a reference point for comparison by training the baseline neural model using our latvian language dataset. as discussed in the preceding work section, we defined a first baseline for prediction as the accuracy achieved by always assigning the most frequent annotation value in the corpus. however, different from the english dataset, the most frequent value in the latvian dataset is 2 (with a distribution of values @xcite @xcite @xcite ), and the corresponding baseline accuracy is 30.5% the baseline neural model uses 64 hidden units, vocabulary size 100 and a bag-ofwords input representation. using input preprocessed with the genseg segmentation tool gives the highest performance, yielding a total 49% accuracy, which is an 8% improvement over the old model, of which 1.25% can be attributed to the subword segmentation. c1 c2 c3 c4 rm bm accuracy 48.8 48.4 47.5 46.9 42.2 30.5 in this paper, we have proposed a new neural network-based model for frustration intensity prediction for customer messages in the context of conversations with customer support representatives, as well as a new dataset of such conversations in latvian, used to train and evaluate the model performance. a baseline model used for comparison employs a bag-of-words representation as input, constructed on the basis of vocabulary that is dynamically built during the training phase. the words selected for inclusion in the vocabulary are the ones that have the least standard deviation for annotated frustration intensity values. our proposed method differs, first and most importantly, by including also features constructed on the basis of non-lexical means of expression, and, secondly, by performing close-tomorphological segmentation as a preprocessing step to make vocabulary construction more coherent. we also examined a few other methods of input processing, namely, removal of diacritics, and unifying popular variations in spelling, but these did not yield any improvements in results, even when used together. performance was assessed by performing a leave-one-out cross exhaustive cross-validation, that is, by computing accuracy (as percentage of correct predictions) obtained after training the model on all data except one entry using this one entry for prediction, and repeating this process for all the entries in turn, averaged across five runs. we compare against a similar neural model that only uses lexical features as input. while such a model achieves a 10% improvement over the baseline accuracy obtained by always predicting the median frustration rating of the dataset, our proposed model achieves an improvement in accuracy of 18% over the baseline and 8% over the old model. we have conducted ablation studies to evaluate the contribution attributable to input preprocessing using close-to morphological segmentation, and also adjusted hyperparameters, and found that the segmentation is responsible for approximately 1.25% of the improvement. in addition, we have presented a new dataset in latvian, that contains dialogs between users and customer support specialists. the dataset in total has 283 dialogs with 688 user turns and 531 customer support representative turns, with each dialog containing no less than two user turns separated by a support representative turn, and with all user turns manually annotated for frustration intensity level. this dataset was used for training and assessing the reference neural network-based model for frustration prediction and its improved version. nicholson, joy, kazuhiko takahashi, and ryohei nakatsu. "emotion recognition in speech using neural networks." neural computing & applications 9, no. 4 (2000): 290-296. alm, cecilia ovesdotter, dan roth, and richard sproat. "emotions from text: machine learning for text-based emotion prediction." in proceedings of human language technology conference and conference on empirical methods in natural language processing, pp. 579-586. 2005. ekman, paul. "an argument for basic emotions." cognition & emotion 6, no. 3-4 (1992): 169-200. yao, yuanlin, shuwei wang, ruifeng xu, bin liu, lin gui, qin lu, and xiaolong wang. "the construction of an emotion annotated corpus on microblog text."