Using Respiration for Enhancing Human-Robot Dialogue (2024)

Abstract: This paper presents the development and capabilities of a spoken dialogue robot that uses respiration to enhance human-robot dialogue. By employing a respiratory estimation technique that uses video input, the dialogue robot captures user respiratory information during dialogue. This information is then used to prevent speech collisions between the user and the robot and to present synchronized pseudo-respiration with the user, thereby enhancing the smoothness and engagement of human-robot dialogue.

## introduction for spoken dialogue robots to be effectively used in various scenarios, it is crucial for human-robot dialogue to be as natural as human-human dialogue. in human-human dialogue, communication occurs not only through verbal language but also through non-verbal cues. thus, incorporating non-verbal information is essential for enhancing the naturalness of human-robot dialogue. previous research has shown that integrating non-verbal cues such as nodding and body movements into robots can improve dialogue fluency @xcite , confirming the benefits of including non-verbal information in dialogue robots. our focus is on a specific type of non-verbal information: respiration. we believe that integrating respiratory information can significantly enhance human-robot dialogue, as respiration is intimately connected to speech. our research has demonstrated that respiratory information is effective in predicting user speech onset @xcite . based on this finding, we developed a spoken dialogue system that uses user respiratory information to predict user speech onset, helping to prevent speech collisions in human-robot dialogue @xcite . furthermore, to ensure accurate capture of user respiratory informa-tion in real-world dialogue settings, we have implemented a respiratory estimation method that uses video input. this method employs the first-ever deep learning model to provide a robust estimation of the respiratory waveform, even in the presence of speech movements, marking an improvement over an existing method @xcite . building on these developments, we created a spoken dialogue robot that uses respiration. the dialogue robot estimates user respiratory waveform values and uses them for enhancing human-robot dialogue. initially, the dialogue robot predicts user speech onset and initiates dialogue responses only when user utterances are not predicted, thereby preventing speech collisions. this approach is designed to facilitate smoother human-robot dialogue by ensuring that the conversation flows without interruptions. a previous study in human-robot dialogue has confirmed that avoiding speech collisions contributes to smoother turn-taking @xcite . secondly, the dialogue robot presents synchronized pseudo-respiration with user respiration. this approach is designed to enhance the dialogue robot's impression. research in robots has shown that the presentation of pseudo-respiration can enhance the robot's impression @xcite . furthermore, a study in human-human dialogue has demonstrated that synchronized respiration during turn-taking leads to smoother transitions between speakers (rochet-capellan and fuchs, 2014). to confirm the impact of these approaches on human-robot dialogue, we conducted a dialogue experiment in which 50 participants each interacted individually with the dialogue robot. we used actual respiratory waveform values obtained from a respiratory measurement device as user respiration @xcite . preliminary analysis, conducted after the initial report, indicates that adjusting the timing of robot speech onset using user speech prediction effectively reduces speech ## system overview our dialogue robot comprises a respiratory waveform estimator, a voice activity predictor, a communication robot, and a spoken dialogue system. figure 1 provides an overview of these components and their arrangement. respiratory waveform estimator: we developed a respiratory waveform estimator using a deeplearning model comprising 3dcnn-convlstm, which is robust against speaker motion @xcite . this estimator uses rgb video frames of a user as input to estimate the user's respiratory waveform amplitude and gradient at the time of the final frame. the model was trained using vrwidataset foot_0 . the estimated values are then transmitted to both the voice activity predictor and the communication robot. voice activity predictor: we developed a voice activity predictor using a single-layer long short-term memory (lstm) network that processes es-timated respiratory waveform values as input. this model predicts whether user speech will occur within the next 200 ms during non-speaking periods. it was trained on a dataset created using the vrwidataset, which extracts data from user non-speaking intervals. this dataset pairs the respiratory waveform values over a specific period with the user's voice activity occurring 200 ms later. the prediction results are then transmitted to the spoken dialogue system. ## use case our dialogue robot is designed to demonstrate the effectiveness of integrating respiratory information into human-robot dialogue. ## conclusion and future work in pursuit of facilitating smooth and engaging human-robot dialogue, we have developed a spoken dialogue robot that uses respiration. this dialogue robot employs a respiratory estimation method using video input to capture user respiratory information, which serves two primary purposes: predicting user speech onset to prevent speech collisions in dialogues, and presenting pseudo-respiration synchronized with the user's respiration. these approaches are expected to enhance the smoothness and engagement of human-robot dialogue. while adjusting speech timing contributes to smoother human-robot dialogue, reducing the number of robot utterances could detract the naturalness of the dialogue. to address this concern, future work will focus on incorporating non-verbal cues such as gaze into the voice activity prediction model, aiming to enhance its accuracy and ensure the dialogue robot does not unnecessarily remain silent. additionally, accurate capture of user respiration is essential for the prediction in natural dialogue, so we will also work on developing a more robust respiratory waveform estimation method. furthermore, we aim to develop a pseudorespiration presentation that considers robot utterances, preventing a decrease in robot utterances while maintaining pseudo-respiration. through these enhancements, we aim to use respiratory information more effectively, achieving more natural human-robot dialogue.