Semantic Differentiation in Speech Emotion Recognition: Insights from Descriptive and Expressive Speech Roles (2025)

Abstract: Speech Emotion Recognition (SER) is essential for improving human-computer interaction, yet its accuracy remains constrained by the complexity of emotional nuances in speech. In this study, we distinguish between descriptive\ semantics , which represents the contextual content of speech, and expressive\ semantics , which reflects the speaker’s emotional state. After watching emotionally charged movie segments, we recorded audio clips of participants describing their experiences, along with the intended emotion tags for each clip, participants’ self-rated emotional responses, and their valence/arousal scores. Through experiments we show that descriptive semantics align with intended emotions, while expressive semantics correlate with evoked emotions. Our findings inform SER applications in human-AI interaction and pave the way for more context-aware AI systems.

## introduction the ability to accurately detect and interpret emotions in speech is vital for developing intelligent systems capable of natural and empathetic humancomputer interactions. speech emotion recognition (ser) has gained significant traction in recent years, driven by applications ranging from virtual assistants to mental health monitoring @xcite . despite these advancements, ser faces persistent challenges due to the complex and multi-dimensional nature of emotions, which often intertwine with contextual and speakerspecific factors. traditional approaches to ser have largely focused on acoustic features, such as pitch, energy, and spectral properties, to infer emotional states @xcite @xcite ; kwon * equal contributions † corresponding author @xcite . while effective to some extent, these methods often overlook the semantic content of speech, which can provide crucial contextual information. with the advances in natural language processing, it has become increasingly feasible to analyze the semantic aspects of speech for emotion recognition @xcite . however, the interplay between semantic roles and emotional expression remains underexplored. specifically, the distinction between intended emotions elicited by a stimulus and evoked emotions experienced by the speaker is rarely addressed, leaving a critical gap in the field. this paper introduces a novel framework to address this gap by distinguishing two types of semantic roles in speech. we hypothesize that descriptive semantics captures scenario-specific content, such as the narrative or context described in the speech. in contrast, expressive semantics reflects the speaker's subjective emotional stance, shaped by their personal experiences and delivery style. in our framework, descriptive segments are expected to align with the intended emotion of the stimulus (the target emotion the video was designed to elicit), while expressive segments are expected to align with the evoked emotion (the participant's self-reported experience). this mapping allows us to distinguish stimulus-driven affect from speakerspecific affect, thereby addressing a critical gap in prior ser research that often assumes a single ground-truth label. this semantic distinction is particularly important in settings where it is essential to understand not only what happened -the contextual content of speech -but also how it was felt -the speaker's emotional state and tone. such an understanding has practical implications for applications like emotion-aware ai systems, educational tools, and interactive entertainment, where both the content and emotional delivery of speech play key roles in creating engaging and effective figure 1 : data collection and algorithm workflow: participants watched six videos eliciting specific emotions and provided speech descriptions, emotion ratings, and valence/arousal scores. speech data were transcribed, segmented into descriptive and expressive semantics, and used to train models for three tasks: predicting intended emotions (task-1), evoked emotions (task-2), and valence/arousal (task-3). to validate our hypothesis, we collected a dataset comprising emotionally evocative movie clips to elicit a specific emotion. participants watched the videos and provided ratings for the actual evoked emotions, alongside valence and arousal scores, creating a robust foundation for analysis. our methodology to uncover the distinct relationships between semantic roles and intended versus evoked emotions involves three key steps: speech transcription with automatic speech recognition (asr), semantic segmentation with llms, and emotion prediction with fine-tuned text classifiers/regressors. this work makes the following contributions: importantly, our work goes beyond simply predicting emotion labels from participants' descriptions. by explicitly segmenting speech into descriptive and expressive roles, we quantify how different semantic functions relate to stimulus-intended versus self-experienced emotions. this role-based separation provides a principled way to reconcile discrepancies between intended and evoked affect and offers interpretable insights that are not available from standard text-only or audio-only models. our findings have significant implications for designing more accurate and context-aware emotion recognition systems, with potential applications in virtual assistants, customer service, and mental health support. by bridging the gap between semantics and emotion, this research advances the state-of-the-art in ser and sets the stage for future exploration of semantic roles in emotional ai systems. ## related work emotions are complex psychological and physiological responses to salient events, involving bodily sensations, expressive behaviors, and cognitive evaluations @xcite . various linguistic features, including prosody, lexical choice, and sentence structure, play a role in the perception and expression of emotions @xcite @xcite . speech emotion recognition (ser) models aim to detect emotional states from speech using acoustic, textual, or multimodal signals. with the advancement of llms and automatic speech recognitions (asr), text-based emotion classification has seen improved accuracy @xcite @xcite . acoustic-based emotion detectors have also progressed using acoustic feature extractors, such as opensmile @xcite or audio embedding models, such as wav2vec @xcite and hubert @xcite , which embed paralinguistic cues such as pitch, tempo, and energy into speech representations @xcite @xcite @xcite . multi-modal approaches, which combine speech, facial expressions, and physiological signals, have also become increasingly prominent in recent years @xcite @xcite @xcite . emotion elicitation via multimedia stimuli (e.g. short film clips) is a common technique in ser to induce targeted emotions (e.g., sadness, joy, fear) @xcite . these movie-based emotion elicitation techniques have applications in various fields, including e-health monitoring and human-computer interaction @xcite . the stimuli are selected and validated through self-report and physiological measures @xcite @xcite . while these methods control for the intended emotional target, they do not always account for the evoked emotion the speaker experiences and expresses. prior work such as siedlecka and denson (2019) reviewed these paradigms in detail, but focused primarily on affect induction rather than the emotional content of participants' verbal responses. in our work, we analyze speech collected after stimulus exposure and study how intended and evoked emotions are reflected in participants' spoken descriptions. in doing so, we explore a novel distinction between semantic roles in language-namely, whether a speaker is being descriptive (e.g., summarizing the movie) or expressive (e.g., conveying their own reaction)-and how these roles align with different emotion types. many ser datasets have been developed. in acted speech datasets, such as iemocap @xcite and savee (jackson and haq, 2014), actors are recruited to read sentences or act in scenes that portray different emotions. in spontaneous speech datasets, such as msp-podcast (lotfian and busso, 2017), msp-conversation (martinez-lucas et al., 2020), samaine @xcite , and recola @xcite , and elicited speech datasets, such as lssed @xcite , baum-1 @xcite , and enter-face @xcite , audios are recorded in a freely speaking environment or with emotion elicitation methods. speech is then annotated by a third party (perception-of-other). however, these datasets focus on one emotion label per speech and do not distinguish different types of emotions. to this end, emo-db @xcite and iemocap @xcite analyzed emotional evocative sentences and perception-of-other in acted speech. most similar to us, muse (jaiswal and bara, 2020) collects speech following emotional video stimuli and reports both self-reported and intended emotion annotations. while similar in structure, our work uniquely interprets the relationship between stimulus-intended and self-reported emotions through a semantic lens, enabling direct analysis of misalignment between the two emotion types. furthermore, some recent studies in nlp have explored emotion elicitation and manipulation in conversational settings @xcite @xcite . while our study does not model conversational interactions, our semantic framework may offer insights into these settings by helping to identify when emotional influence is being attempted or received. for example, expressive speech segments may signal internal affective states, while descriptive segments may reflect contextual awareness or narrative framing. these distinctions could inform models of emotion transfer and regulation in human-computer dialogue. our contribution lies in bridging the gap between stimulus-based emotion elicitation and the actual emotions conveyed by participants in speech. by segmenting utterances according to their semantic roles and analyzing how different roles align with either intended or evoked emotions, we propose a novel way to interpret emotional speech beyond traditional modality-based or label-based approaches. while prior ser studies have emphasized either acoustic or multimodal representations, our work suggests that semantic structure in language -accessible only through text -offers a distinct and interpretable signal for differentiating between types of emotion. movie clip tag scene description duration validation source the blair witch project @xcite fear final scene when screaming intensifies, man standing facing the wall and camera falls. ## dataset the block diagram in figure 1 summarizes our data collection, task definitions, and methodology, which we will elaborate on here and in the next section. data collection was carried out in person at inspire laboratory of the school of psychology at university of ottawa. the experiment procedure was approved by the research ethics board of university of ottawa. the study included 97 student participants aged 18 to 27 (@xmath0.9, @xmath1.5). the majority were women (81 women, 15 men, and 1 non-binary), and most participants were native english speakers (65 spoke english as their first language, 12 spoke french, and 20 spoke other languages). the sample was ethnically diverse, comprising 16 asian, 20 black/african, 7 hispanic/latino, 1 indigenous, 15 mixed/multiple ethnicities, 33 white/caucasian, and 5 participants identifying as other. our study focused on the six basic emotions identified by @xcite as the target emotions in our experimental setup: sadness, fear, joy, dis-gust, surprise, and anger. two movie clips for each emotion were sourced from film stimuli in the existing literature and validated in our pilot study. the twelve movie clips used in the study and their metainformation are listed in their effectiveness was validated in a pilot study with 25 participants before the final data collection. in the main study, participants watched six emotional video clips, one from each emotion category. to re-establish baseline levels of valence and arousal, the presentation of each emotional clip was preceded by a neutral video clip. to further mitigate potential carryover effects between conditions, a two-minute rest period was inserted between each neutral-emotional clip sequence, during which one of six still images was displayed on the computer screen. all video clips and still images were presented in random order to minimize potential sequence effects. the collected dataset consists of 97× 6 entries, with five elements: 1) speech: a 30-second audio recording of the participant's verbal response to the following instruction: ## tasks we define three tasks to examine the relationship between semantic types and emotion recognition. to determine the most predictive semantic type for each task, we experimented with three different inputs: full transcriptions, descriptive semantic segments, and expressive semantic segments. task-1: classification of intended emotion involves classifying the intended emotion associated with each video based on participants' speech. task-2: classification of evoked emotion involves classifying participant-reported evoked emotions, which are subjective and may include multiple emotions simultaneously. while evoked emotions often include the intended emotion, individual differences can lead to variations. this task is framed as a multi-label classification problem, where each emotion (on a scale of 0 to 6) is binarized based on whether it is evoked or not. task-3: regression of valence and arousal predicts participants' self-reported valence and arousal ratings, which provide a two-dimensional representation of emotional states. ## methodology as depicted in figure 1 , our methodology consists of three sequential steps: speech recognition, semantic segmentation, and emotion prediction. step-1: automatic speech recognition -we used whisper @xcite , an automatic speech recognition model, to transcribe the participants' speech data into text. step-2: semantic segmentation -we used gpt-4o (openai, 2023) to extract descriptive and expressive segments from the transcription obtained in step 1. the prompt is given in ## experiments 6.1 implementation details for step-1, automatic speech recognition, we used 'whisper-large-v3' foot_0 , a state-of-the-art system asr model known for its robustness across diverse accents and noise conditions. we manually reviewed transcriptions in the development set, consisting of 33 × 6 audio transcriptions. whisper achieved a the user will provide a paragraph describing their feelings towards a particular movie, delimited with "'####"'. your task is to segment the paragraph into two parts according to the type of content: descriptive segments and expressive segments. descriptive segments refer to elements or clauses that provide factual or narrative information about the movie content without explicitly reflecting personal emotions or opinions. expressive segments refer to elements or clauses that convey personal feelings, attitudes, or opinions. these segments reflect individual reactions, emotions, and perceptions, or the intensity of these emotions. the two parts (descriptive segments and expressive segments) can overlap, but all clauses of the given paragraph must be contained in at least one of the two parts. output your answer in the following format: <@xmath2 for step-3, emotion prediction, we fine-tuned different classifiers/regressors, including bert (devlin, 2018), roberta @xcite , and de-berta @xcite . different text semantics identified in step-2 are used as inputs to the models. for emotion classification (tasks 1 and 2), we used the text embeddings from the models and applied a standard classification head with a softmax activation function to predict categorical emotions. for regression (task 3), we modified the models by replacing the classification head with a fully connected layer that outputs a single continuous value, trained with mean squared error (mse) loss to predict valence and arousal scores. this approach follows standard practice in adapting transformer encoders for regression tasks @xcite @xcite . data are split on participants' level, with 1/3 of partici-how often do participants experience the intended emotion conveyed by the videos? 96.63% how frequently do participants feel emotions other than the intended one? 89.39% how often is the intended emotion rated as the highest by participants? ## conclusion this study introduces a novel framework for speech emotion recognition (ser) by distinguishing between semantic roles in speech. by leveraging llms' zero-shot capabilities in text segmentation, we tackle a previously difficult challenge. to our knowledge, this is the first work to segment speech into two semantic roles, expressive and descriptive content, to enable more fine-grained and nuanced emotion detection. our findings reveal that descriptive semantics are more predictive of intended emotions, while expressive semantics are more closely aligned with evoked emotions and their valence and arousal dimensions. this differentiation can inform future research in emotion detection. in some contexts, it might be more useful to instruct users and guide them toward only one of these modes of expressing emotions. in other applications, it might be more suitable to leave it to the users to express their emo-tions in a mixture of expressive and descriptive modes. the llms can then be used to segment the speech and use the segments depending on the predictive goals. this approach enhances the development of more accurate and context-aware emotion recognition systems, with applications in mental health, virtual assistants, and customer service.