Modeling Emotion Dynamics in Song Lyrics with State Space Models (2023)

Abstract: Most previous work in music emotion recognition assumes a single or a few song-level labels for the whole song. While it is known that different emotions can vary in intensity within a song, annotated data for this setup is scarce and difficult to obtain. In this work, we propose a method to predict emotion dynamics in song lyrics without song-level supervision. We frame each song as a time series and employ a State Space Model (SSM), combining a sentence-level emotion predictor with an Expectation-Maximization (EM) procedure to generate the full emotion dynamics. Our experiments show that applying our method consistently improves the performance of sentence-level baselines without requiring any annotated songs, making it ideal for limited training data scenarios. Further analysis through case studies shows the benefits of our method while also indicating the limitations and pointing to future directions.

## introduction music and emotions are intimately connected, with almost all music pieces being created to express and induce emotions @xcite . as a key factor of how music conveys emotion, lyrics contain part of the semantic information that the melodies cannot express @xcite . lyrics-based music emotion recognition has attracted increasing attention driven by the demand to process massive collections of songs automatically, which is an important task for streaming and media service providers @xcite @xcite . vanilla emotion recognition studies in natural language processing (nlp) assume the text instance expresses a static and single emotion @xcite @xcite . however, emotion is non-static and highly correlated with the contextual information, making the single-label assumption too simplistic in dynamic scenarios, not just in music @xcite but also in other domains such as conversations @xcite . figure 1 shows an example of this dynamic behavior, where the intensities of three different emotions vary within a song. accurate emotion recognition systems should ideally generate the full emotional dynamics for each song, as opposed to simply predicting a single label. a range of datasets and corpora for modeling dynamic emotion transitions has been developed in the literature @xcite @xcite @xcite , but most of them do not use song lyrics as the domain and assume discrete, categorical labels for emotions (either the presence or absence of one emotion). to the best of our knowledge, the dataset from @xcite is the only one that provides full fine-grained emotion intensity annotations for song lyrics at the verse 1 level. the lack of largescale datasets for this task poses a challenge for traditional supervised methods. while previous work proposed methods for the similar sequencebased emotion recognition task, they all assume the availability of some levels of annotated data at training time, from full emotion dynamics @xcite to coarse, discrete documentlevel labels @xcite . the data scarcity problem motivates our main research question: ''can we predict emotion dynamics in song lyrics without requiring annotated lyrics? @xcite in this work, we claim that the answer is affirmative. to show this, we propose figure 1 : an illustration of emotion dynamics of a song in the lyricsemotions dataset of @xcite . note the intensities of each emotion vary from verse to verse within the song. a method consisting of two major stages: (1) a sentence or verse-level regressor that leverages existing emotion lexicons, pre-trained language models and other sentence-level datasets, and (2) a state space model (ssm) that constructs a full song-level emotional dynamics given the initial verse-level scores. intuitively, we treat each verse as a time step and the emotional intensity sequence as a latent time series that is inferred without any song-level supervision, directly addressing the limited data problem. to the best of our knowledge, this scenario was never addressed before either for song lyrics or other domains. to summarize, our main contributions are: the categorical model assumes a fixed set of discrete emotions that can vary in intensity. emotions can overlap but are assumed to be separate ''entities'' from each other, such as anger, joy, and surprise. taxonomies using the categorical model include ekman's basic emotions @xcite ), plutchik's wheel of emotions @xcite , and the occ model @xcite . the dimensional models place emotions in a continuous space: the vad (valence, arousal, and dominance) taxonomy of @xcite is the most commonly used in nlp. in this work, we focus on the ekman taxonomy for purely experimental purposes, as it is the one used in the available data we employ. however, our approach is general and could be applied to other taxonomies. ## method we propose a two-stage method to predict emotion dynamics without requiring annotated song lyrics. the first stage is a verse-level model that predicts initial scores for each verse, where we use a hybrid approach combining lexicons and sentence-level annotated data from a different domain ( § 3.1). the second stage contextualizes these scores in the entire song, incorporating them into an lg-ssm trained in an unsupervised way ( § 3.2). ## experiments our experiments aim to evaluate the method proposed to predict the emotional dynamics of song lyrics without utilizing any annotated lyrics data. we introduce datasets, lexicon resources, and the evaluation metric used ( §4.1), and discuss the implementation details and experiment settings of the verse-level model ( §4.2) and the song-level model ( §4.3). ## results and analysis in this section, we first compare the results of our lexicon-based, learning-based and hybrid methods at the verse level ( § 5.1). we then provide the results of the song-level models and investigate the impact of the initial predictions from verse-level models, ssm parameters, and parameter optimization ( § 5.2). we additionally show the qualitative case analysis results to understand our model's abilities and shortcomings ( § 5.3). finally, we compare the results of supervised and unsupervised methods on lyricsemotions ( § 5.4). ## conclusion and future work this paper presents a two-stage bertlex-ssm framework for sequence-labeling emotion intensity recognition tasks, especially in label-scarce scenarios. combining the contextualized embeddings with static word embeddings and then modeling the initial predicted intensity scores as a state space model, our method can utilize context-sensitive features with external knowledge and capture the emotional dynamic transitions. experimental results show that our proposed bertlex-ssm effectively predicts emotion intensities in the lyrics without requiring annotated lyrics data. our findings and analysis point to a range of directions for future work: domain adaptation. while our method could apply any general verse-level model, including a pure lexicon-based one, in practice, we obtained the best results by leveraging annotated sentencelevel datasets. this naturally leads a domain discrepancy: in our particular case, between news and lyrics domains. given that unlabeled song lyrics are relatively easy to obtain, one direction is to incorporate unsupervised domain adaptation techniques @xcite to improve the performance of the verse-level model. semi-supervised learning (similar to @xcite ) is another promising direction, although methods would need to be modified to incorporate the continuous nature of the emotion labels.