Improving the Generalizability of Text-Based Emotion Detection by Leveraging Transformers with Psycholinguistic Features (2022)

Abstract: recent years, there has been increased interest in building predictive models that harness natural language processing and machine learning techniques to detect emotions from various text sources, including social media posts, micro-blogs or news articles. Yet, deployment of such models in real-world sentiment and emotion applications faces challenges, in particular poor out-of-domain generalizability. This is likely due to domain-specific differences (e.g., topics, communicative goals, and annotation schemes) that make transfer between different models of emotion recognition difficult. In this work we propose approaches for text-based emotion detection that leverage transformer models (BERT and RoBERTa) in combination with Bidirectional Long Short-Term Memory (BiLSTM) networks trained on a comprehensive set of psycholinguistic features. First, we evaluate the performance of our models within-domain on two benchmark datasets GoEmotion (Demszky et al., 2020) and ISEAR (Scherer and Wallbott, 1994). Second, we conduct transfer learning experiments on six datasets from the Unified Emotion Dataset (Bostan and Klinger, 2018) to evaluate their out-of-domain robustness. We find that the proposed hybrid models improve the ability to generalize to out-of-distribution data compared to a standard transformer-based approach. Moreover, we observe that these models perform competitively on in-domain data.’

## introduction emotions are a key factor affecting all human behavior, which includes rational tasks such as reasoning, decision making, and social interaction @xcite @xcite . although emotions seem to be subjective by nature, they appear in objectively derivable ways in texts. text-based emotion detection (henceforth tbed) is a branch of sentiment analysis that aims to extract textual features to identify associations with various emotions such as anger, fear, joy, sadness, surprise, etc. tbed is a rapidly developing interdisciplinary field that brings together insights from cognitive psychology, social sciences, computational linguistics, natural language processing (nlp) and machine learning @xcite @xcite . tbed has a wide range of real-world applications, from healthcare @xcite , recommendation systems @xcite , empathic chatbot development @xcite , offensive language detection @xcite , social data analysis for business intelligence @xcite , and stock market prediction @xcite . the differentiation of emotions and their classification into specific groups and categories is a subfield of affective research and has yielded several theories and models @xcite @xcite @xcite . the grouping of models for the classification of emotions generally differs according to whether emotions are conceived as discrete/categorical or as dimensional. categorical models of emotions, like ekman's six basic emotions (anger, disgust, fear, joy, sadness, and surprise) @xcite , assume physiologically distinct basic human emotions. plutchik's wheel of emotion @xcite is another categorical model that assumes a set of eight discrete emotions expressed in four opposing pairs (joy-sadness, anger-fear, trust-disgust, and anticipation-surprise). dimensional emotion models, like the circumplex model of @xcite , groups affective states into a vector space of valence (corresponding to senti-ment/polarity), arousal (corresponding to a degree of calmness or excitement), and dominance (perceived degree of control over a given situation). current approaches to tbed take the advantage of recent advances in nlp and machine learning, with deep learning techniques achieving state-of-the-art performance on benchmark emotion datasets (see @xcite for recent reviews). however there still remains the issue of out-of-domain generalizability of the existing emotion detection models. the way emotions are conveyed in texts may differ from domain to domain, reflecting differences in topics, communicative goals, target audience, etc. this makes the deployment of such models in real-world sentiment and emotion applications difficult. the importance of this issue has been increasingly recognized in the tbed literature. for example, @xcite emphasize that "[j]ournalists ideally tend to be objective when writing articles, authors of microblog posts need to focus on brevity", and that "emotion expressions in tales are more subtle and implicit than, for instance, in blogs". to support future transfer learning and domain adaptation work for tbed, the authors constructed a unified, aggregated emotion detection dataset that encompasses different domains and annotation schemes. in this work, we contribute to the improvement of the generalizability of emotion detection models as follows: we build hybrid models that combine pre-trained transformer language models with bidirectional long short-term memory (bilstm) networks trained, to our knowledge, on the most comprehensive set of psycholinguistic features. we evaluate the performance of the proposed models in two ways: first, we conduct within-corpus emotion classification experiments (training on one corpus and testing on the same) on two emotion benchmark datasets, goemotion @xcite and isear @xcite , to show that such hybrid models outperform pretrained transformer models. second, we conduct transfer learning experiments on six popular emotion classification datasets of the unified emotion dataset @xcite to show that our approach improves the generalizability of emotion classification across domains and emotion taxonomies. the remainder of the paper is organized as follows: in section 2, we briefly review recent related work on tbed. then, in section 3, we present popular benchmark datasets for emotion detection. section 4 details the extraction of psycholinguistic features using automated text analysis based on a sliding window approach. in section 5, we describe our emotion detection models, and in section 6, we present our experiments and discuss the results. finally, we conclude with possible directions for future work in section 7. ## related work in this section, we focus on previous tbed research conducted on two popular benchmark datasets (goemotions, isear) to compare the performance of our models with state-of-the-art emotion recognition models, as well as previous attempts to improve generalizability using transfer learning techniques. current work on tbed typically utilizes a variety of linguistic features, such as word or character n-grams, affect lexicons, and word embeddings in combination with a supervised classification model (for recent overviews see, @xcite @xcite . while earlier approaches relied on shallow classifiers, such as a naive bayes, svm or maxent classifier, later approaches increasingly relied on deep learning models in combination with different word embedding methods. for example, @xcite proposed an emotion detection model based on the use of long short-term memory (lstm) and convolutional neural network (cnn) mediated through the use of a level of attention in combination with different word embeddings @xcite . in experiments performed on the isear dataset, @xcite proposed a text emotion distribution learning model based on a lexiconenhanced multi-task convolutional neural network (lmt-cnn) to jointly solve the tasks of text emotion distribution prediction and emotion label classification. the lmt-cnn model is an end-to-end multi-module deep neural network that utilizes semantic information and linguistic knowledge to predict emotion distributions and labels. based on comparative experiments on nine commonly used emotion datasets, @xcite showed that the lmt-cnn model can outperform two previously introduced deep-neural-network-based models: textcnn, a convolutional neural network for text emotion classification @xcite and mt-cnn @xcite , a multi-task convo-lutional neural network model that simultaneously predicts the distribution of text emotion and the dominant emotion of the text (see for the recently introduced goemotions dataset, @xcite already provided a strong baseline for modeling emotion classification by fine-tuning a bert-base model. their model achieved an average f1-score of 64% over an ekman-style grouping into six coarse categories. ? conducted comparative experiments with additional transformer-based models -bert, distil-bert, roberta, xlnet, and electra @xcite ) -on the goemotions dataset. as in the case of isear, the best performance was achieved by roberta, with an f1-score of 49% on the full goemotions taxonomy (28 emotion categories). previous tbed work has also proposed combinations of different approaches. for example, @xcite proposed a hybrid model that combines emotion keywords in a sentence using an emotional keyword dictionary with a knowledgebased artificial neural network that uses domain knowledge. to our knowledge, however, almost no tbed research has investigated hybrid models that combine transformer-based models with (psycho)linguistic features (see, however, de bruyne et al. 2021, for an exception in dutch). this is surprising, as such an approach has been successfully applied in related areas, for example personality prediction @xcite . the available research aimed at improving the generalizability of transformer-based models using transfer learning techniques has so far focused on demonstrating that training on a large dataset of one domain, say reddit comments, can contribute to increasing model accuracy for different target domains, such as tweets and personal narratives. specifically, using three different finetuning setups -(1) finetuning bert only on the target dataset, (2) first finetuning bert on goemotions, then perform transfer learning by replacing the final dense layer, and (3) freezing all layers besides the last layer and finetuning on the target dataset -, demszky et al. (2020) showed that the goemotions dataset generalizes well to other domains and different emotion taxonomies in nine datasets from the unified emotion dataset @xcite . ## datasets we conduct experiments on a total of eight datasets. the within-domain experiments are performed on two benchmark corpora: the goemotions dataset @xcite and the international survey on emotion antecedents and reactions (isear) dataset @xcite . goemotions is the largest available manually annotated dataset for emotion prediction. it consists of 58 thousand reddit comments, labeled by 80 human raters for 27 emotion categories plus a neutral category. while 83% of the items of the dataset have received a single label, goemotions is strictly speaking a multilabel dataset, as raters were free to select multiple emotions. the dataset has been manually reviewed to remove profanity and offensive language towards a particular ethnicity, gender, sexual orientation, or disability. the isear dataset is a widely used benchmark dataset consisting of personal reports on emotional events written by 3000 people from different cultural backgrounds. it was constructed by collecting questionnaires answered by people that reported on their own emotional events. it contains a total of 7,665 sentences labeled with one of seven emotions: joy, fear, anger, sadness, shame, guilt and disgust. the transfer-learning experiments are conducted on six benchmark datasets from unified emotion dataset @xcite ) that were chosen based on their diversity in size and domain: (1) the affectivetext dataset @xcite consists of 1,250 news headlines. the annotation schema follows ekman's basic emotions, complemented by valence. it is multi-label annotated via expert annotation and emotion categories are assigned a score from 0 to 100. (2) the crowdflower dataset consists of 39,740 tweets annotated via crowdsourcing with one label per tweet. the dataset was previously found to be noisy in comparison with other emotion datasets @xcite . (3) the dataset electoral-tweets @xcite targets the domain of elections. it consists of over 100,000 responses to two detailed online questionnaires (the questions targeted emotions, purpose, and style in electoral tweets). the tweets are annotated via crowdsourcing. (4) the stance sentiment emotion corpus ssec @xcite is an annotation of 4,868 tweets from the semeval 2016 twitter stance and sentiment dataset. it is annotated via expert annotation with multiple emotion labels per tweet following plutchik's fundamental emotions. (5) the twitter emotion corpus tec @xcite consists of 21,011 tweets. the annotation schema corresponds to ekman's model of basic emotions. they collected tweets with hashtags corresponding to the six ekman emotions: #anger, #disgust, #fear, #happy, #sadness, and #surprise, therefore it is distantly single-label annotated. (6) the emotion-stimulus dataset @xcite has 1,549 sentences with their emotion analysed. the set of annotation labels comprises of ekman's basic emotions with the addition of shame. (7) the isear ued dataset that is part of the unified emotion dataset has 5,477 sentences with single emotion annotations. this dataset is a filtered version of the original isear dataset described above. @xcite filter and keep the texts with the labels anger, disgust, joy, sadness and fear for the unified emotion dataset. ## sentence-level measurement of psycholinguistic features the datasets were automatically analyzed using an automated text analysis (ata) system that employs a sliding window technique to compute sentencelevel measurements (for recent applications of this tool across various domains, see @xcite for fake news detection, @xcite for predicting human affective ratings) and wiechmann et al. (2022) for predicting eye-moving patterns during reading). we extracted a set of 435 psycholinguistic features that can be binned into four groups: (1) features of morpho-syntactic complexity (@xmath0, (2) features of lexical richness, diversity and sophistication (@xmath1, (3) readability features (@xmath2, and (4) lexicon features designed to detect sentiment, emotion and/or affect (@xmath3. tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic pcfg parsing were performed using stanford corenlp @xcite . the group of morpho-syntactic complexity features includes (i) surface features related to the length of production units, such as the average length of clauses and sentences, (ii) features of the type and frequency of embeddings, such as number of dependent clauses per t-unit or verb phrases per sentence and (iii) the frequency of particular structure types, such as the number of complex nominals per clause. this group also includes (iv) information-theoretic features of morphological and syntactic complexity based on the deflate algorithm @xcite . the group of lexical richness, diversity and sophistication features includes six different subtypes: (i) lexical density features, such as the ratio of the number of lexical (as opposed to grammatical) words to the total number of words in a text, (ii) lexical variation, i.e. the range of vocabulary as manifested in language use, captured by text-size corrected type-token ratio, (iii) lexical sophistication, i.e. the proportion of relatively unusual or advanced words in a text, such as the number of words from the new general service list @xcite , (iv) psycholinguistic norms of words, such as the average age of acquisition of the word @xcite and two recently introduced types of features: (v) word prevalence features that capture the number of people who know the word @xcite and (vi) register-based n-gram frequency features that take into account both frequency rank and the number of word ngrams (@xmath4, 5]). the latter were derived from the five register subcomponents of the contemporary corpus of american english (coca, 560 million words, @xcite : spoken, magazine, fiction, news and academic language (see @xcite , for details see e.g.). the group of readability features combines a word familiarity variable defined by a prespecified vocabulary resource to estimate semantic difficulty along with a syntactic variable, such as average sentence length. examples of these measures include the fry index @xcite or the smog @xcite . the group of lexiconbased sentiment/emotion/affect features was derived from a total of ten lexicons that have been successfully used in personality detection, emotion recognition and sentiment analysis research: (1) the affective norms for english words (anew) (bradley and lang, 1999), (2) the anew-emo lexicons @xcite , ( 3 ) depechemood++ @xcite , (4) the geneva affect label coder (galc) @xcite , ( 5 ) general inquirer @xcite , ( 6 ) the liwc dictionary @xcite , ( 7 ) the nrc word-emotion association lexicon @xcite , (8) the nrc valence, arousal, and dominance lexicon @xcite , ( 9 ) senticnet @xcite , and (10) the sentiment140 lexicon @xcite . ## modeling approach we construct a total of five models: (1) a fine-tuned bidirectional encoder representations from transformers (bert) model, (2) a fine-tuned roberta model (robustly optimized bert pre-training approach), (3) a bidirectional neural network classifiers trained on sentence-level measurements of psycholinguistic features described in section 3.1, and ( 4 ) and ( 5 ) two hybrid models integrating bert and roberta predictions with the psycholinguistic features. we train all models in a multi-label classification setup. for the withindomain evaluation of the models on the goemotions dataset, we follow the procedure specified in @xcite : that is, we filtered out emotion labels selected by only a single annotator. the 93% of the original were randomly split into train (80%), dev (10%) and test (10%) sets. these splits are identical to those used by demszky et al.. in the transfer learning setting geared to show that our modeling approach improves generalization across domains and taxonomies, we perform experiments on each of the six emotion benchmark datasets presented in section 3 using four approaches: with/without finetuning on target dataset and with/without the inclusion of the label 'neutral'. the performance of these models is evaluated using 5 times repeated 5-fold crossvalidation using a 80/20 split to counter variability due to weight initialization. we report performance metrics averaged over all runs. all models are implemented using pytorch @xcite . unless specifically stated otherwise, we use 'bceloss' as our loss function, 'adamw' as optimizer, with learning rate 2 × 10 -5 and weight decay of 1 × 10 -5 ## results the models were evaluated using accuracy, precision, recall and f1 scores as the performance metrics. the results of the within-domain classification experiments on the goemotion and isear datasets are shown in an overview of the results of the out-of-domain experiments is presented in ## conclusion this paper proposed approaches for text-based emotion detection that leverage transformer models in combination with bidirectional long short-term memory networks trained on a comprehensive set of psycholinguistic features. the results of transfer learning experiments performed on six outof-domain emotion datasets demonstrated that the proposed hybrid models can substantially improve model generalizability to out-of-distribution data compared to a standard transformer-based model. moreover, we found that these models perform competitively on in-domain data. in future work, we intend to extend this line of work to dimensional emotion models as well as to models that jointly solve the tasks of emotion label classification and text emotion distribution prediction.