Emosical: An Emotion-Annotated Musical Theatre Dataset (2024)

Abstract: This paper presents Emosical, a multimodal open-source dataset of musical films. Emosical comprises video, vocal audio, text, and character identity paired samples with annotated emotion tags. Emosical provides rich emotion annotations for each sample by inferring the background story of the characters. To achieve this, we leverage the musical theatre script, which contains the characters’ complete background stories and narrative contexts. The annotation pipeline includes feeding the speaking character, text, global persona, and context of the dialogue and song track into a large language model. To verify the effectiveness of our tagging scheme, we perform an ablation study by bypassing each step of the pipeline. The ablation results show the usefulness of each component in generating accurate emotion tags. A subjective test is conducted to compare the generated tags of each ablation result. We also perform a statistical analysis to find out the global characteristics of the collected emotion tags. Emosical would enable expressive synthesis and tagging of the speech and singing voice in the musical theatre domain in future research. Emosical is publicly available at https://github.com/gillosae/emosical.

## introduction as a fundamental aspect of human experience, recognizing emotion requires a depth of understanding in the multi-domains. with the advances in deep learning techniques, various approaches delve into detecting emotion through expressed emotion from different modalities @xcite . studies aim to detect emotion through the domain of text @xcite , speech @xcite , and facial expressions @xcite . furthermore, approaches to integrate multiple modalities @xcite have been introduced. while those techniques have advanced the understanding of human emotions in different domains, emotion recognition still remains a challenge. first, it is hard to capture ground-truth emotion, which is expressed through both linguistic and non-linguistic elements. this complexity requires multi-modal data to capture the emotional context fully, but integrating and synchronizing such diverse inputs presents technical difficulties @xcite . although some models are trained using recorded multi-modal datasets @xcite , creating these datasets is resourceintensive and lacks the scalability and diversity needed to fully capture the complexity of human emotions. additionally, human emotions are complex and subtle, with the inherent ambiguity of expressed emotional cues and variations in cultural expressions @xcite . thus, existing models struggle to recognize emotions from expressed features at a fine-grained level, mostly clustering emotions into coarse categories @xcite . we suspect that knowledge of the underlying personality or situation makes it easier to infer one's emotion more accurately and finely, which is a capability missing in most datasets. we propose that theatre is a particularly effective medium for addressing these challenges. theatre naturally encompasses multi-modal emotional expressions since actors and directors use various techniques such as dialogue, music, lighting, and stage design to communicate a wide range of emotions to the audience @xcite . these emotions are typically expressed with an emphasis for greater impact @xcite , making it easier to capture and analyze them effectively. also, the theatre has the unique property of containing a 'script' for the play, which enables the inference of each character's storyline and personality. in this view, theatre contains a diverse range of expressed emotions conveyed through multiple modalities that are deeply rooted in each character's narrative context, making it a suitable medium for emotion research. in response, we build emosical, an emotionannotated theatre dataset. specifically, we select musical theatre for our dataset in order to contain singing samples. there is no theatre dataset specifically curated for machine learning purposes. it might be due to theatre's inherent complexity, which combines spoken natural language, audio, and visual elements, making it hard to create complex recordings manually. therefore, we design an annotation pipeline that infers emotion from the narrative by leveraging the script from each theatre. our annotation pipeline aims to analyze and annotate the narrative's emotions as automatically as possible. thus, we crawl musical theatre films and apply them as pipeline input. we require theatre film video with the timestamp and speaker identity aligned srt file as the inputs. in the pipeline, we feed the character, text, global persona, and context of the dialogue and song track into an llm to obtain the corresponding emotion tag per sample. our annotation pipeline captures emotion effectively by capturing the narrative context through fed inputs. in musical theatre, some prominent songs tend to be reprised and presented multiple times throughout the act, conveying different emotional nuances. figure 1 shows an example; the number 'healing incantation' of 'tangled' is the case, which emerges once at an introductory moment, once at a highly-elated scene, and lastly at the ending part of the movie. in the figure, the three reprised songs have distinctly different emotions, clearly indicating diverse emotional expressions. 'healing incantation' is reprised triple times in the movie. even though they all have the same lyrics our tagging pipeline tags corresponding singing emotions well by inferring emotions from the context and character's persona. through our pipeline, even though the song consists of the same lyrics, the resulting tags are different due to different scene contexts fed to obtain the tags. this pipeline has another advantage that it applies to building some types of emotion-paired datasets, which are hard to develop. for instance, there is no public singing data annotated with emotions aside from @xcite . unlike speech, singing demands attention to nuances like pitch, tone, and emotional delivery @xcite . even though singing is a powerful medium that conveys expressive emotional expression, it is hard to record a singing voice sample with specific emotions manually. however, our approach enables emotional inference on any kind of data, including singing -under the condition that the identity's background context is given. through text, video, and vocal-aligned samples with rich emotion annotations, we aim for emosical to be primarily used to understand the relationship between theatre's multi-modal characteristics and emotions. additionally, we hope the dataset could also be used for tasks such as emotion tagging and emotional synthesis for each modality; a baseline system for the former is developed and evaluated as an example. to this end, emosical provides detailed and fine-grained emotion annotation for every short segment. unlike emotion-annotated datasets @xcite , which tend to annotate emotions in broad groups over long segments, our dataset with dense emotion tags annotated to short data samples allows for more precise and temporal studies of emotional changes. we summarize our contributions as follows: ## related works ## dataset overview. ## conclusion we presented a novel dataset, emosical, the first open-source multimodal dataset specifically curated for musical films with comprehensive emotion annotations. by integrating video, audio, text, and character identity with emotion tags derived from a detailed narrative context, emosical provides a rich resource for advancing research in emotion recognition, synthesis, and tagging in the musical theatre domain. our dataset leveraged a novel annotation pipeline, incorporating global persona, scene context, visual description, and dialogue or lyrics to generate nuanced emotion tags using a large language model (llm). through statistical analysis and a series of ablation studies, we demonstrated the effectiveness of our tagging scheme. our subjective evaluations further validated the precision and reliability of our annotations. additionally, we proposed a baseline tagging model for emotion recognition in singing voices, setting a foundation for future research in this area. emosical opens up new avenues for exploring the interplay between various modalities in conveying emotions and can serve as a valuable resource for developing more emotionally resonant systems. future work may include expanding the dataset to encompass more diverse genres and languages, refining the emotion tagging pipeline, and exploring its applications in various multimodal emotion recognition and synthesis tasks. we believe emosical can contribute to further research in multimodal understanding of emotion expressions in musical theatre.