Multi-Emotion Classification for Song Lyrics (2021)

Abstract: Song lyrics convey a multitude of emotions to the listener and powerfully portray the emotional state of the writer or singer. This paper examines a variety of modeling approaches to the multi-emotion classification problem for songs. We introduce the Edmonds Dance dataset, a novel emotion-annotated lyrics dataset from the reader’s perspective, and annotate the dataset of Mihalcea and Strapparava (2012) at the song level. We find that models trained on relatively small song datasets achieve marginally better performance than BERT (Devlin et al., 2018) fine-tuned on large social media or dialog datasets.

## introduction text-based sentiment analysis has become increasingly popular in recent years, in part due to its numerous applications in fields such as marketing, politics, and psychology @xcite @xcite . however, the vast majority of sentiment analysis models are built to identify net positive or negative sentiment rather than more complex, ambiguous emotions such as anticipation, surprise, or nostalgia @xcite . as a result, current models usually fail to portray the coexistence of multiple emotions within a text sample, resulting in limited characterization of a human's true emotions. songs are often created to elicit complex emotional responses from listeners, and thus are an interesting area of study to understand nuanced emotions @xcite . this paper examines a variety of approaches to address the multi-emotion classification problem. we aim to build an emotion classification model that can detect the presence of multiple emotions in song lyrics with comparable accuracy to the typical inter-annotator agreement for textbased sentiment analysis (70-90%) @xcite @xcite . building such a model is especially challenging in practice as there often exists considerable disagreement regarding the perception and interpretation of the emotions of a song or ambiguity within the song itself @xcite . there exist a variety of high-quality text datasets for emotion classification, from social media datasets such as cbet @xcite and tec @xcite to large dialog corpora such as the dailydialog dataset @xcite . however, there remains a lack of comparable emotionannotated song lyric datasets, and existing lyrical datasets are often annotated for valence-arousal affect rather than distinct emotions @xcite . consequently, we introduce the edmonds dance dataset foot_0 , a novel lyrical dataset that was crowdsourced through amazon mechanical turk. our dataset consists of scalar annotations for the 8 core emotions presented by @xcite , with annotations collected at the song level and from the reader's perspective. we find that bert models trained on out-ofdomain data do not generalize well to song lyrics and have lower f1 scores than naive bayes classifiers for emotions such as disgust and fear. however, bert models trained on small lyrical datasets achieve marginally better performance, despite indomain datasets being orders of magnitude smaller than their counterparts. we also find that surprise has significantly lower inter-annotator agreement and test accuracy than other core emotions. ## related work a multitude of models and techniques have been explored for song emotion classification. both @xcite and @xcite found that fea-ture extraction from lyrics improves emotion classification performance. researchers have trained naive bayes, hmm, svm, clustering, and random forest models on lyrical and sometimes audio features to predict emotion in songs @xcite @xcite @xcite . deep learning frameworks have also been widely utilized for song emotion classification, ranging from cnns and lstms @xcite to transformer-based models such as bert and elmo @xcite . multiple researchers have taken a multi-modal approach to emotion prediction. @xcite , introduced a novel corpus of both music and lyrics, and achieved promising results when using both musical and lyrical representations of songs in emotion classification. similarly, @xcite found an increase in 4-class emotion prediction accuracy from 46.6 to 57.1 percent when incorporating lyrics into models trained on audio. however, audio data can lead to problematic bias in emotion classification. @xcite explored the presence of emotion stereotyping in certain genres, and found that heavy metal and hip-hop music were perceived to have more negative emotions than pop music with matched lyrics. @xcite also found that emotional responses to an audio sample of a song could be predicted by stereotypes of the culture with which the song's genre was associated. additionally, @xcite found that violent lyrical passages were seen to be significantly more negative when represented as rap songs rather than country songs. @xcite validated fried's findings through multiple studies in which participants believed that identical lyrics were more offensive when portrayed as rap rather than country music. lyrics are paramount for the accurate prediction of emotion in music. @xcite transformed song lyrics into psychological feature vectors using a content analysis package and concluded that song lyrics alone can be used to generate promising, human-comprehensible classification models. @xcite found that audio features did not always outperform lyric features for mood prediction, and that combining lyric and audio features does not necessarily improve mood prediction over simply training on lyrics features. in later research, @xcite found that lyrics features significantly outperformed au-dio features in 7 of 18 mood categories, while audio features outperformed lyrical features in only one. research is split regarding crowdsourced emotion annotation quality; while @xcite achieved strong results through crowdsourcing labels, @xcite found crowd labels to sometimes not even be in agreement with themselves. surprise is an emotion that is especially difficult to model @xcite , less frequent (oberländer and klinger, 2018), and is sometimes divided into positive and negative surprise @xcite . ## model implementation we chose naive bayes as our first baseline emotion classification model due to its widespread applications in text classification and sentiment analysis @xcite . given its robustness to outliers and ability to deal with imbalanced data @xcite , a random forest baseline model was also implemented. lastly, we utilized a most frequent sense (mfs) baseline model, given its strong performance in word sense disambiguation tasks and its applications to emotion classification @xcite . we trained our naive bayes model on bag-of-words features and our random forest model on transformed feature vectors which were generated from our textual datasets using the nrc hashtag emotion lexicon @xcite ; see appendix for further details. to improve upon emotion classification quality, we also explored more complex models. due to its ability to generate powerful contextualized word embeddings and its state-of-the-art results in numerous language understanding tasks @xcite , the bert base uncased architecture was fine-tuned for multi-emotion classification from the text of song lyrics. bert base consists of 12 transformer blocks, a hidden size of 768, 12 self-attention heads, and an additional output layer which we used for fine-tuning. foot_1 ## evaluation we trained separate bert models for each emotion on the original and augmented cbet datasets, and tested their performance on the edmonds dance and mihalcea/strapparava datasets. we then compared these results with those of our baseline naive bayes, random forest, and most frequent sense models. to compare emotion prediction accuracy across multiple text corpora, we also trained bert models on the tec and dailydialog datasets, and tested them on our lyrical datasets. we found that bert models trained on the cbet, tec, and dailydialog datasets did not generalize well to lyrical data. while models for joy and sadness improved upon the performance of baseline classifiers, models for disgust and fear performed worse than our naive bayes baseline. furthermore, data augmentation techniques improved the performance of our baseline naive bayes model, but did not significantly increase bert model accuracy. to compare in-domain model accuracy with our out-of-domain results, we trained and tested bert models on the edmonds dance and mihalcea/strapparava datasets, and vice versa. models trained and tested on lyrical datasets had marginally better accuracy and f1 scores than out-of-domain models for anger, joy, and sadness. given the much smaller sizes of lyrical datasets compared to their counterparts, as well as the differences in song genre and annotation perspective across lyrical datasets, our findings suggest a significant advantage in using in-domain data to train models for complex emotion classification of songs. finally, all models performed poorly when classifying surprise, and f1 scores for anger, disgust, and fear remained consistently low across models, suggesting a steep hierarchy of difficulty regarding emotion classification. inter-annotator agreement was much lower for surprise than other emotions, and none of our models were able to accurately predict the presence of surprise in song lyrics. our work implies that surprise is unique from the perspective of emotion classification. ## conclusion in this paper we explore a variety of approaches to the multi-emotion classification problem for songs. we introduce the edmonds dance dataset, a novel lyrical dataset annotated for emotion at the song level and from the reader's perspective. we find that emotion classification of song lyrics using state-of-the-art methods is difficult to accomplish using out-of-domain data; bert models trained on large corpora of tweets and dialogue do not generalize to lyrical data for emotions other than joy and sadness, and are outperformed by naive bayes classifiers on disgust and fear. on the other hand, models trained on song lyrics achieve comparable accuracy to models trained on out-of-domain data, even when lyrical datasets are orders of magnitude smaller than their counterparts, have been aggregated from line to song level, have been annotated from different perspectives, and are composed of different genres of music. our findings underscore the importance of using in-domain data for song emotion classification.