Analysis of Voice Activity Detection Errors in API -based Streaming ASR for Human-Robot Dialogue (2025)

Abstract: In human-robot dialogue systems, streaming automatic speech recognition (ASR) services (e.g., Google ASR) are often utilized, with the microphone positioned close to the robot’s loudspeaker. Under these conditions, both the robot’s and the user’s utterances are captured, resulting in frequent failures to detect user speech. This study analyzes voice activity detection (VAD) errors by comparing results from such streaming ASR to those from standalone VAD models. Experiments conducted on three distinct dialogue datasets showed that streaming ASR tends to ignore user utterances immediately following system utterances. We discuss the underlying causes of these VAD errors and provide recommendations for improving VAD performance in human-robot dialogue.

## introduction several embodied robots capable of speech interaction have been developed @xcite . in this situation, since such robots are embodied, automatic speech recognition (asr) is performed without the user wearing a headset microphone, meaning the microphone is not close to the user's mouth. instead, the robot usually uses its built-in loudspeaker and a microphone placed nearby. with advancements in asr technology, even researchers who do not specialize in asr can easily use it. several previous studies have compared the performance of off-the-shelf asr, providing valuable information for researchers and developers outside the speech recognition community, e.g., @xcite . one of the simplest ways to use asr is through streaming-based asrs accessed via apis. by using these, asrs do not need to be downloaded or installed in advance, making them very convenient. in this case, because the asr is streaming-based, both voice activity detection (vad) and speech recognition are user utterance failed to be detected performed simultaneously on the server side. vad @xcite ) is a crucial frontend technology for spoken dialogue systems. its role is to detect the active speech segments from the input signals captured by microphones. vad is primarily used to determine the boundaries of a user utterance, facilitating turn-taking in dialogue @xcite @xcite . however, if the system misses the user utterance, causing the dialogue to break down. thus, accurately detecting user speech segments is also important for asr @xcite . errors in failing to detect user utterances are significant problems for both turn-taking and asr and therefore must be avoided. in this paper, we demonstrate situations where system fails to detect user utterances occur in robot dialogue. specifically, we focus on scenarios where (1) a streaming-based asr is used, and (2) the robot's loudspeaker and microphone are positioned in close proximity. in such cases, the robot's voice is also picked up by the microphone along with the user's voice. this leads to frequent failures in vad when using streaming-based asr trained on single-speaker data, as depicted in figure 1 . vad error result in the robot ignoring the user utterance. this causes the robot to remain silent without taking its turn, making it difficult for general users to understand the robot's status, leading to significant frustration. therefore, our research aims to provide insights for human-robot interaction (hri) researchers and practitioners to achieve robust vad. this study addresses the following two research questions: to answer these research questions, we conducted multiple analyses. as experimental validation, we demonstrate vad error (i.e., failures of vad to detect user utterances) in a streamingbased asr system using three datasets with different microphone configurations. we also compare the performance to a case where vad is performed separately. on the basis of these results, we discuss the appropriate system configurations for conveniently building conversational robots. ## related works several studies have analyzed the performance and errors of publicly available asrs. as asr can be increasingly used by non-experts, it is useful to analyze the performance and issues of various asrs. @xcite showed that the google api's asr tends to ignore disfluencies, i.e., non-fluent words. @xcite compared the asrs of microsoft, ibm, and google, investigating their robustness against disfluencies and overlaps. @xcite analyzed the characteristics of several asrs, demonstrating that performance degrades in domains requiring specialized vocabulary, as well as in noisy environments. @xcite investigated the impact of accents, i.e., various native and non-tative accents in english, on the performance of off-theshelf asrs. the whisper model was found to be particularly sensitive to variations in accents. these studies focus on analyzing asr performance. however, it is also helpful to analyze the causes, especially in situations involving interaction with embodied robots. error analysis in a intention signal channel conversation speaker a's actions addressee b's actions a is proposing to b b is considering a's proposal a is conveying an utterance to b b is understanding a's utterance a is presenting a signal to b b is identifying a's signal a is executing behavior for b b is attending to behavior from a convenient streaming-based asr, rather than in segmented speech files, is also helpful. particularly, a failure in vad causes more damage to the dialogue than simple asr errors, as it means the system does not recognize that the user has spoken. therefore, we focus on vad and examine situations when its errors occur in dialogues with embodied robots, where a robot utterance can be picked up by the microphone along with the user utterance. a model that predicts future user utterance segments, rather than detecting the end of the user utterance segment, was proposed. voice activity projection (vap) @xcite is a model that predicts the future voice activity of two speakers, on the basis of raw audio input. the model requires two channels of recorded speech as input. since our study targets a situation in which a dialogue system is used in a simple recording environment, and thus targets speech segment detection on the basis of one-channel recorded speech, vap is not included in the comparison in this study. ## target of analysis vad is a crucial component in spoken dialogue systems. according to the action ladder @xcite illustrated in fig. 2 , joint actions between interlocutors at the signal level are established when the addressee successfully identifies the signal emitted by the speaker. on the basis of this, joint actions at higher levels, such as the intention level and conversation level, can be established. errors in vad lead to two main issues in spoken dialogue systems: corresponding to these two issues, we will analyze vad performance both frame-wise and turnwise, as illustrated in fig. 3 . furthermore, we examined the extent to which speech recognition accuracy improves due to speech segment detection. ## data and models this section describes the dataset and vad models used for the analysis in this study. ## results of analyses the analysis results for ex.1 and ex.2 are summarized in section 5.1. more detailed frame-level analyses are conducted in sections 5.2 and 5.3. in section 5.2, we analyze the impact of speech segment detection errors on speech recognition. in section 5.3, we investigate at which points during user utterances the detection of speech segments fails.