Estimating the Emotional Valence of Interlocutors Using Heterogeneous Sensors in Human-Human Dialogue (2024)

Abstract: Dialogue systems need to accurately understand the user’s mental state to generate appropriate responses, but accurately discerning such states solely from text or speech can be challenging. To determine which information is necessary, we first collected human-human multimodal dialogues using heterogeneous sensors, resulting in a dataset containing various types of information including speech, video, physiological signals, gaze, and body movement. Additionally, for each time step of the data, users provided subjective evaluations of their emotional valence while reviewing the dialogue videos. Using this dataset and focusing on physiological signals, we analyzed the relationship between the signals and the subjective evaluations through Granger causality analysis. We also investigated how sensor signals differ depending on the polarity of the valence. Our findings revealed several physiological signals related to the user’s emotional valence.

## introduction most current user-adaptive dialogue systems rely on text or speech to estimate the user's state and generate appropriate responses. however, the user's state that can be inferred solely from text or speech is limited. consequently, there has been active research on estimating the user's state from multimodal data, particularly focusing on user emotions and engagement through the analysis of facial expressions, gestures, and gaze @xcite . in recent years, the application of physiological signals in dialogue systems has also gained popularity. for example, studies have been conducted to estimate a user's sentiment and emotions from physiological signals @xcite @xcite . however, these studies have typically utilized a limited range of sensors and have not dealt with the real-time nature of the user's mental state, which is essential for dialogue systems. therefore, in this study, we used heterogeneous sensors to collect a variety of data during humanhuman dialogues, including speech, video, physiological signals, gaze information, and motion information. after each dialogue, for each time step of the data, interlocutors conducted immediate subjective evaluations of their emotional valence while watching recordings of the dialogues. as an analysis, we used granger causality analysis to investigate the relationship between the information obtained from the heterogeneous sensors and the subjective evaluation annotations. we also conducted a statistical test to examine how sensor signals differ depending on the polarity of valence. various sensor signals were collected, but in this paper, we focus on physiological signals, as these data are believed to be closely related to mental states @xcite . our key contributions in this work are as follows: ## related work several multimodal dialogue corpora have been constructed that include information such as the interlocutor's movements and gaze in addition to speech. for example, humaine @xcite is a multimodal dialogue corpus covering various topics aimed at eliciting user emotions. the iemocap dataset @xcite is a script-based human-human dialogue dataset containing speech, video, and facial information. semaine @xcite is a corpus containing dialogues between computer graphics (cg) agents with different personalities and human subjects. d64 @xcite ) is a multi-party dialogue corpus designed to capture the natural reactions and emotions of the interlocutors. the physiological signals are measured and quantified by sensors for physiological phenomena (such as heartbeat, brain waves, pulse, respiration, and perspiration) and can deal with the real-time state of the interlocutor. several multimodal dialogue corpora have been constructed that include the physiological signals of the interlocutor in a dialogue. for example, recola @xcite is a human dialogue dataset that includes physiological signals during a collaborative dialogue task. electrocardiogram (ecg) and electrodermal activity (eda) are utilized as physiological signals in recola. the pegconv dataset @xcite comprises discussion dialogues and includes galvanic skin response (gsr) and photoplethysmography (ppg) as physiological signals during the dialogue. hazumi @xcite is a multimodal dialogue corpus containing dialogues between a human and a cg agent. the physiological signals include eda, blood volume pulse (bvp), skin temperature (temp), and heart rate (hr) data @xcite . although several corpora have been constructed in this way, none of the corpora contain data that comprehensively includes movement, gaze, and a variety of physiological signals. moreover, to the best of our knowledge, there has been no research on estimating the real-time user state required by dialogue systems from sensor signals in dialogues. ## data collection the data were collected in two periods, with the first beginning in december 2023 and the second in february 2024. to distinguish the two datasets, we used the year and month of data collection for naming: "data2312" for the data collected in december 2023 and "data2402" for the data collected in february 2024. in these two sets of data collection experiments, a total of 40 interlocutors (21 male, 19 female), all native japanese speakers, participated. they were recruited from the general public by a recruiting agency, and each participated in only one data collection experiment. two interlocutors were paired into one group and engaged in 10-minute dialogues on three different topics: "chit-chat", "narrative", and "discussion". immediately after each dialogue, interlocutors annotated their subjective evaluations related to emotional valence while watching the recordings of the dialogue. a detailed summary of the collected dataset is provided in both data collection experiments were conducted in the same sequence: pre-experiment questionnaire administration, sensor placement and attachment, dialogue and annotation conduction, sen-sor removal, and follow-up questionnaire administration. the dialogue and annotation conduction process was repeated three times for the three topics. for each topic, the following sequence was repeated: dialogue conduction, post-dialogue questionnaire administration, and subjective evaluation. the experiments were approved by the ethics committee of our institution. in the following subsections, we describe in detail the multimodal data and heterogeneous sensors, the three dialogue topics, the questionnaires, and the subjective evaluation of dialogues. ## data analysis human emotional mental states, such as happiness and sadness, are formed through the brain's processing of information from three sources: 1) information from the body (e.g., hr, sweating, and other physiological states), 2) information from the external world (e.g., visual and auditory input, etc.), and 3) memories stored in the brain @xcite . in our collection of multimodal data, the physiological signals obtained from embraceplus, shimmer3 gsr+, and pupil core (e.g., eda, ppg, pupil diameter) captured the interlocutors' physiological states (i.e., information from the body), while subjective evaluations annotated the emotional valence of the interlocutors. in this study, eda and bvp (collected from em-braceplus), ppg (collected from shimmer3 gsr+), and pupil diameter (collected from pupil core) were used as physiological signals. we first performed data preprocessing on these signals for subsequent analysis. we then performed granger causality analysis to examine the relationship between these physiological signals and subjective evaluations of emotional valence, i.e., whether these physiological signals can be used to predict subjective evaluations. finally, we analyzed the differences between these physiological signals under different polarities of valence. ## results in this section, we present the results of the granger causality analysis and the differences in the means of physiological signals between "positive" and "negative" valence. ## conclusion and future work in this study, we collected dialogue data containing comprehensive multimodal data and subjective evaluations at each time step during the dialogue using heterogeneous sensors. through our analysis of the relationship between physiological signals and emotional valence using the granger causality analysis, we identified several physiological signals that could be useful for predicting real-time emotional valence. we also clarified how physiological signals differ depending on the "positive" or "negative" polarity of the valence. however, several limitations of our study should be acknowledged. first, the relatively small sample size limits the statistical power of our findings and reduces the generalizability of the results to a larger population. second, the imbalanced ratio of positive to negative valence samples may potentially lead to biased conclusions about the relationship between physiological signals and emotional valence. future research needs to apply methodologies for analyzing imbalanced and small sample size data. moreover, we plan to expand our analysis to include sensor signals, linguistic information, questionnaires about personality traits, and impressions of the dialogue, in addition to physiological signals. we will also use the information from the sensors to predict emotional valence in real-time. ultimately, our goal is to achieve a dialogue system capable of estimating and appropriately responding to the user's mental state in real-time.