Integrating Physiological, Speech, and Textual Information Toward Real-Time Recognition of Emotional Valence in Dialogue (2025)

Abstract: Accurately estimating users’ emotional states in real time is crucial for enabling dialogue systems to respond adaptively. While existing approaches primarily rely on verbal information, such as text and speech, these modalities are often unavailable in non-speaking situations. In such cases, non-verbal information, particularly physiological signals, becomes essential for understanding users’ emotional states. In this study, we aimed to develop a model for real-time recognition of users’ binary emotional valence (high-valence vs. low-valence) during conversations. Specifically, we utilized an existing Japanese multimodal dialogue dataset, which includes various physiological signals, namely electrodermal activity (EDA), blood volume pulse (BVP), photoplethysmography (PPG), and pupil diameter, along with speech and textual data. We classify the emotional valence of every 15-second segment of dialogue interaction by integrating such multimodal inputs. To this end, time-series embeddings of physiological signals are extracted using a self-supervised encoder, while speech and textual features are obtained from pre-trained Japanese HuBERT and BERT models, respectively. The modality-specific embeddings are integrated using a feature fusion mechanism for emotional valence recognition. Experimental results show that while each modality individually contributes to emotion recognition, the inclusion of physiological signals leads to a notable performance improvement, particularly in non-speaking or minimally verbal situations. These findings underscore the importance of physiological information for enhancing real-time valence recognition in dialogue systems, especially when verbal information is limited.

## introduction understanding and responding to users' emotional states is a fundamental capability for user-adaptive dialogue systems. such systems must be able to detect and adapt to users' emotional states in real time to foster natural and engaging interactions. while prior work has made considerable progress in emotion recognition by leveraging verbal information @xcite , notably text and speech, these approaches often fall short in non-speaking or minimally verbal contexts, which are common in real-world interactions. in these cases, relying solely on verbal information can hinder the system's ability to maintain emotional awareness and responsiveness. to address this limitation, non-verbal modalities, especially physiological signals, offer valuable cues about users' internal states. physiological signals, such as eda and bvp, have been shown to reflect changes in emotional arousal and valence @xcite @xcite . these signals are continuous, language-independent, and inherently real-time, making them suitable for supplementing verbal inputs during emotionally relevant but verbally sparse moments in dialogue. in this study, we tackle binary emotional valence recognition (high-valence vs. low-valence) in conversational settings by integrating physiological signals with speech and text. as a preliminary step toward developing real-time emotion recognition systems, we develop a model to classify the emotional valence of each 15-second dialogue segment using a japanese multimodal dialogue dataset that includes multiple physiological signals alongside speech recordings and textual transcripts. to this end, we adopt a multimodal recognition framework that combines (1) time-series embeddings of physiological signals extracted via a self-supervised encoder, (2) speech features from a pre-trained japanese hubert model, and (3) text embeddings from a pre-trained japanese bert model. the modality-specific representations are then fused via a feature fusion mechanism and used for downstream classification. our findings demonstrate the effectiveness of integrating physiological signals with verbal information for emotional valence recognition and provide an assessment of the individual contributions of eda, bvp, ppg, and pupil diameter. the results underscore the potential of physiological signals to enhance the emotional intelligence of dialogue systems, enabling robust performance even during verbally limited or silent phases of interaction. recognizing the emotional states of speakers during dialogue is essential for achieving smooth and adaptive communication. toward this goal, a wide range of approaches leveraging textual, speech, and facial expression information have been proposed @xcite . recent studies have explored multimodal fusion to improve emotion recognition performance. for example, hazarika et al. (2020) proposed misa, a framework that integrates text, audio, and visual information via contextual inter-modal attention. zadeh et al. ( 2018 ) introduced the cmu-mosei dataset along with the dynamic fusion graph for modeling cross-modal interactions. @xcite developed dialoguernn, which tracks speaker states and conversational context using multimodal features. however, most of these approaches primarily rely on verbal and visual modalities. meanwhile, physiological signals, particularly cardiac-related signals such as electrocardiograms (ecg), have emerged as promising modalities for emotion recognition. traditional methods extract handcrafted features such as heart rate variability (hrv) and apply classifiers such as svms or random forests. more recently, the dynamic nature of physiological data has motivated the application of time-series modeling techniques, including lstmbased recurrent networks and transformer architectures, to capture the temporal dependencies and improve prediction accuracy @xcite . in addition, self-supervised learning methods, such as ts2vec @xcite and contrastive learning frameworks @xcite , have been proposed to learn rich physiological representations without extensive labeled data. several studies have explored the integration of physiological signals with other modalities to improve emotion recognition performance in humancomputer interactions. for instance, @xcite proposed a multimodal framework combin-ing eeg and speech signals, demonstrating that eeg can significantly enhance speech-based emotion classification. similarly, @xcite examined eda and visual features to assess users' binary sentiment states in dialogue, achieving an accuracy of 63.2% and emphasizing the improvements gained through physiological signal integration. despite these advancements, many prior studies continue to rely heavily on verbal information and often assume access to complete utterances or entire conversational segments. this overlooks the challenges of real-time emotion recognition during ongoing interactions, especially in non-speaking or minimally verbal situations. to address this limitation, we propose a multimodal recognition framework that integrates physiological signals, speech, and textual information. by predicting emotional valence over short, fixed time intervals, our approach would enable robust emotion recognition, even in conversational environments with limited linguistic information. ## dataset to investigate multimodal recognition of emotional valence, we used an existing japanese humanhuman multimodal dialogue dataset that we previously created @xcite , which contains multimodal information recorded by heterogeneous devices. in this study, we focus on leveraging physiological signals and verbal information for emotional valence recognition. among the available modalities in the dataset (text, speech, video, physiological signals, and motion information), we selected physiological data (eda, bvp, ppg, and pupil diameter), speech data, and text data for training the recognition models. a detailed description of the selected data is provided in ## multimodal recognition framework to recognize emotional valence from both verbal and non-verbal cues, we developed a multimodal recognition framework that integrates text, speech, and physiological data, as illustrated in fig. 1 . ## experiment to evaluate the effectiveness of different input modalities in the recognition of emotional valence, we conducted two rounds of classification tasks: unimodal and multimodal recognition: we assessed both unimodal and multimodal models. we conducted unimodal experiments individually for the physiological, speech, and text modalities. subsequently, we evaluated the performance of the multimodal models by combining different pairs of modalities (text + speech, text + physio, speech + physio), and a model incorporating all three modalities (text + speech + physio). for the physiological modality, all physiological signals (eda, bvp, ppg, and pupil diameter) were used as input. ## conclusion and future work in this study, we proposed a multimodal recognition framework for short-time binary emotional valence recognition by integrating physiological signals, speech, and textual information. leveraging a japanese multimodal dialogue dataset, we extracted time-series representations of physiological signals using a self-supervised encoder and combined them with features from pre-trained japanese hubert and bert models. we used a feature fusion mechanism to dynamically integrate modalityspecific embeddings. experimental results demonstrate that incorporating both speech information and physiological signals significantly enhances the recognition performance of emotional valence, especially in scenarios with limited or no verbal duration transcriptions (15s) t s p s + p gt 10% almost all of them are mail-order. ummm... ah.... low low high high high 11% no, there is no such thing. that is true. it is a little embarrassing. low low high high high 4% mm-hmm. mmm-hmm. mm-hmm. high low low low low