Integrating Respiration into Voice Activity Projection for Enhancing Turn-taking Performance (2025)

Abstract: Voice Activity Projection (VAP) models predict upcoming voice activities on a continuous timescale, enabling more nuanced turn-taking behaviors in spoken dialogue systems. Although previous studies have shown robust performance with audio-based VAP, the potential of incorporating additional physiological information, such as respiration, remains relatively unexplored. In this paper, we investigate whether respiratory information can enhance VAP performance in turn-taking. To this end, we collected Japanese dialogue data with synchronized audio and respiratory waveforms, and then we integrated the respiratory information into the VAP model. Our results showed that the VAP model combining audio and respiratory information had better performance than the audio-only model. This finding underscores the potential for improving the turn-taking performance of VAP by incorporating respiration.

## introduction in conversational systems designed for emotional support and customer assistance, it is crucial for the user and the system to engage in smooth and natural dialogues. a key factor in achieving such smooth communication is effective turn-taking, wherein each participant can seamlessly begin and end speaking without awkward interruptions or prolonged silences. in this context, there has been a growing body of research aimed at predicting turntaking behaviors in spoken dialogue between the user and the system @xcite . recently, voice activity projection (vap) has been proposed as a method for more natural turntaking in spoken dialogue @xcite . vap dynamically models voice activities in dyadic interactions by processing the raw audio signals from both speakers, predicting future voice activity in a series of short time windows (at window lengths of 200 ms, 400 ms, 600 ms, and 800 ms within a 2-second horizon). this approach yields a 256-class prediction representing binary voice activity in each of the four time windows for each speaker. in addition, vap defines four evaluation tasks, shift/hold, short/long, shift-prediction, and backchannel-prediction, to assess how effectively the model can predict turn-shifts and backchannels. specifically, shift/hold tests the model's ability to predict which speaker will take the next turn during mutual silence; short/long tests the ability to predict at its onset whether a speaker's utterance will be a short backchannel or a longer utterance; shift-prediction tests the ability to predict whether a turn-shift will occur during active speech; backchannel-prediction tests the ability to predict future backchannels. various extensions of vap have been explored, including the incorporation of prosodic information, gaze, and gestures @xcite , the extension of multilingual data @xcite , and real-time predictions @xcite . in this work, we aim to further enhance vap by integrating respiratory information, which is a nonverbal cue closely tied to speech production. prior research about respiration has observed the synchronization of respiratory patterns during turntaking (rochet-capellan and fuchs, 2014), as well as behaviors such as speakers taking a quick breath when they wish to continue speaking and next speakers inhaling when the previous speaker finishes speaking @xcite @xcite . these observations have motivated attempts to predict turn continuations, endings, and next-speaker transitions using respiratory signals @xcite . in human-system spoken dialogues, respiration has also been investigated to predict a user's speech onset @xcite , indicating that respiratory information can facilitate smoother turn management. we focus on the turn-taking performance of vap and investigate how integrating respiratory information affects the model's performance in shift/hold, short/long, and shiftprediction tasks. we collected japanese dialogue data containing both audio and respiratory waveforms, and then we integrated the respiratory information into the vap model. our results showed that the vap model combining audio and respiratory information had better performance than the audio-only model. this finding underscores the usefulness of respiratory information for vap turntaking tasks. ## data collection because no publicly available dataset for integrating respiration into vap was available, we collected spoken dialogue data. ## experiments we investigated whether respiratory information can help improve vap performance in turn-taking. ## results the experimental results are shown in ## discussion our results showed that combining respiratory information with audio improves vap performance in turn-taking, especially shift/hold and shift-prediction tasks ( ## limitations and future work although our experiments demonstrated the potential benefits of integrating respiratory information into vap, several limitations remain. first, the amount of data used in this study was relatively small, and participants took part in remote dialogues. to further validate the effectiveness of respiratory information for vap, we plan to collect additional data in more diverse conversational settings. second, we used contact-based respiration sensors to record respiratory waveforms. however, for real-world spoken dialogue systems, it is preferable to measure a user's respiration in a non-contact manner. by combining our approach with noncontact respiratory estimation methods @xcite , which capture users' respiratory information using only an rgb camera, we can eliminate the need for wearable sensors. we will adopt this combined approach to implement vap with integrated respiration in real-world dialogues. third, the method of combining audio and respiratory information in our model was quite simplistic, relying on a straightforward concatenation of features. by improving the model architecture or employing more advanced fusion strategies, it may be possible to more accurately integrate voice and respiratory signals. we will explore these more sophisticated approaches to better leverage respiratory information for vap. finally, although the original vap includes backchannel-prediction, we focused on shift/hold, short/long, and shift-prediction in this study. evaluating the effectiveness of respiratory information on backchannel-prediction remains an important direction for future work and may further clarify the potential of respiratory information. ## conclusion in this work, we explored how respiratory information can be combined with audio to improve voice activity projection (vap). we collected japanese dialogue data with synchronized audio and respiratory waveforms to investigate the efficacy of combining this information for vap. our results indicate that combining audio and respiratory information can improve vap performance in turntaking. this finding underscores the potential value of leveraging respiratory information to enhance the turn-taking performance of vap. we will explore more sophisticated fusion mechanisms that might better integrate respiratory information into vap.