ESP net- SDS : Unified Toolkit and Demo for Spoken Dialogue Systems (2025)

Abstract: Advancements in audio foundation models (FMs) have fueled interest in end-to-end (E2E) spoken dialogue systems, but different web interfaces for each system makes it challenging to compare and contrast them effectively. Motivated by this, we introduce an open-source, user-friendly toolkit designed to build unified web interfaces for various cascaded and E2E spoken dialogue systems. Our demo further provides users with the option to get on-the-fly automated evaluation metrics such as (1) latency, (2) ability to understand user input, (3) coherence, diversity, and relevance of system response, and (4) intelligibility and audio quality of system output. Using the evaluation metrics, we compare various cascaded and E2E spoken dialogue systems with a human-human conversation dataset as a proxy. Our analysis demonstrates that the toolkit allows researchers to effortlessly compare and contrast different technologies, providing valuable insights such as current E2E systems having poorer audio quality and less diverse responses. An example demo produced using our toolkit is publicly available here: https://huggingface.co/spaces/Siddhant/Voice_Assistant_Demo.

## introduction spoken dialogue systems (sds) @xcite are designed to engage in natural and interactive conversations with end users. these systems play a critical role in commercial applications such as voice assistants @xcite and intelligent home devices @xcite . however, this is a challenging task due to the complexity and variability of human-ai communication @xcite . spoken dialogue systems typically consist of multiple modules, including voice activity detection (vad) @xcite , automatic speech recognition (asr) (radford et al., 1 demo video: https://youtu.be/ki_dxwf5qpk 2023; @xcite , natural language understanding (nlu) @xcite and generation (nlg) @xcite , and text-to-speech (tts) synthesis @xcite , each presenting unique challenges. for instance, the asr module must accurately process shorter, spontaneous speech that often contains disfluencies and filler words @xcite . additionally, spoken dialogue systems must understand non-phonemic information @xcite , such as emotions, and generate non-phonemic elements @xcite like laughter to make interactions feel more natural. a critical aspect of these systems is managing the flow of conversation i.e. performing fluent turn-taking without excessive overlapping speech or prolonged silences @xcite . furthermore, effective dialogue systems must be capable of handling long conversational contexts to ensure accurate and context-aware responses @xcite . historically, spoken dialogue systems @xcite have been modeled using a cascaded architecture that incorporates various modules such as asr, nlu, nlg, and tts. recent advances have led to the development of full-duplex end-to-end (e2e) spoken dialogue systems @xcite capable of handling both user input and systemgenerated audio simultaneously. since the main goal of dialogue systems is to enhance the experience of human ai interaction, these dialogue systems are often released with web demos to demonstrate alignment with human conversational expectations. however, there is no unified web interface for users to directly interact with different spoken dialogue systems and compare the relative utility of each system. each spoken dialogue system often has its own web interface with different ways to take user audio as input, different backend computing infrastructure, and sometimes even different approaches to postprocess and show model output. a. input audio from user b. system @xmath0. as the number of spoken dialogue systems and methodologies continues to grow, the need for an open-source toolkit that standardizes the process of creating web interfaces for these systems has become increasingly clear. moreover, there have also been limited efforts to comprehensively evaluate spoken dialogue systems on their conversational capability. as a result, the community lacks an understanding of how best to evaluate these systems. to address this, we introduce the espnet-sds toolkit built on an existing open-source speech processing toolkit espnet @xcite with the aim of providing researchers with easy access to existing cascaded and e2e spoken dialogue methodologies within a unified framework. our goal is to establish a comprehensive opensource standard, enabling researchers to seamlessly integrate existing technologies, compare new ideas, and benchmark their systems against current methodologies. the key contributions of our toolkit are : 1 facilitating the development of gradio foot_0 based demo interfaces, allowing users to interact with and evaluate multiple cascaded and e2e spoken dialogue systems through a standardized platform (fig. 1(c )). 2 providing implementations of automated metrics for submodule-level (e.g., asr, text dialogue, and tts) as well as conversationlevel evaluations (fig. 1(d) ). 3 enabling humanin-the-loop evaluation by including mechanisms for collecting user feedback on the naturalness and relevance of system responses (fig. 1(e) ). 3 our initial experiments on the human-human conversation dataset, switchboard @xcite , highlight the strengths and weaknesses of different spoken dialogue systems, offering actionable insights into the limitations of e2e approaches compared to traditional cascaded pipelines such as current e2e systems generating poorer audio quality and less diverse responses. we hope these findings inspire the development of more robust spoken dialogue systems and evaluation methodologies. the espnet-sds toolkit is publicly available at https://github.com/espnet/ ## related work spoken dialogue systems often include web interfaces that enable users to interact with these systems. however, these interfaces typically feature diverse input/output frontends and backend computing environments, making it difficult to compare systems. this highlights the need for standardized platforms to benchmark systems on a level playing field. similar initiatives in text domain, such as chatbot arena @xcite , have gained significant popularity for benchmarking large language model (llm)-based chatbots using human preference evaluations. inspired by this, tts arena (face, 2024b) has provided a valuable platform for comparing open-source and proprietary tts models. recently, talkarena @xcite was introduced to benchmark audio foundation models (fms) on metrics like the naturalness and coherence of their responses. however, talkarena is currently limited to an audio-in, text-out format and supports only single-turn interactions. to the best of our knowledge, this work presents the first web interface that supports multiple spoken dialogue systems-encompassing both cascaded and e2e methodologies-within a unified framework. these systems are capable of automatically performing turn-taking, generating spoken outputs, and engaging in multi-turn, natural conversations with users. by standardizing the interface and evaluation pipeline, our work not only simplifies the process of testing and comparing dialogue systems but also enhances the accessibility and reproducibility of research in this domain. tab. 1 highlights the unique capabilities and advantages of our proposed web interface compared to prior works. wrappers (python) espnet2/sds #asr modules absasr espnet_asr owsm_asr owsm-ctc_asr whisper_asr #text dialogue modules absllm huggingface_llm #tts modules abstts espnet_tts chat_tts #vad modules absvad webrtc_vad #e2e dialogue modules abse2e miniomni_e2e #interface wrapper for connecting modules espnet_sds_model_ interface template (python) egs2/template/sds1 # evaluation metric --pyscripts/utils/dialog_eval #asr module specific asr_wer #text dialogue module specific llm_metric #tts module specific tts_intelligibility tts_speech_quality #human-in-the-loop evaluation human_feedback #template for gradio demo using interface wrapper & metrics app.py # user guide readme.md recipe (bash) # example egs2/spoken_chatbot_arena/sds1 #launch gradio demo using app.py based on arguments run.sh 3 system design to facilitate benchmarking, the toolkit provides functionality for computing major evaluation metrics at both the sub-task level (e.g., asr, text response generation, and tts) and the overall conversation level (sec. 5). recognizing the importance of human-in-the-loop evaluation, the demo includes functionality to collect human feedback on metrics such as the naturalness and relevance of system responses. additionally, the web interface can store human judgments along with interaction data such as user input audio and system output recordings (sec. 6), but this functionality is disabled by default and intended only for studies with appropriate consent (sec. 10). all evaluation metric functions are implemented under the pyscripts/utils/dialog_eval directory within "template" module (egs2/template/sds1, as shown in fig. 2 ). this module also demonstrates how the interface wrapper and evaluation tools can be utilized to construct a gradio-based spoken dialogue system demo. the app.py file within this module outlines the entire pipeline-from processing user audio input, passing it through the selected spoken dialogue system, to presenting the synthesized output and associated evaluation metrics to the user. additionally, a readme file is included that provides an overview of espnet-sds's features and detailed guidance on using the demo interface. the "template" module serves as a practical starting point for researchers, allowing them to leverage the sds1 template and create customized recipes by passing appropriate arguments to run.sh. these recipes can facilitate comparisons between e2e spoken dialogue systems and traditional cascaded pipelines, as well as test different asr, llm, and tts model combinations within cascaded systems. additionally, they enable on-the-fly comparisons using the supported evaluation metrics. we include an example recipe in egs2/spoken_chatbot_arena/sds1, which can be used to run our demo ( https://huggingface. co/spaces/siddhant/voice_assistant_demo ) locally. the toolkit is designed for ease of extension. when new e2e dialogue systems are introduced, authors or contributors can submit pull requests (pr) to add wrapper functions for their open-source models. corresponding recipes can then be added to pr, integrating the new systems into the unified demo alongside existing models. ## evaluation metrics to evaluate the relative utility of different cascaded spoken dialogue systems, we support the implementation of various automatic evaluation metrics: tts module specific: we utilize the versa foot_4 toolkit @xcite to assess the intelligibility and quality of the system's audio output. intelligibility is evaluated using hypotheses generated by 3 asr models: espnet librispeech asr (watanabe et al., 2018), owsm 3.1 @xcite , and whisper @xcite . audio quality is measured using utmos (utokyo-sarulab system for voicemos 2022 @xcite ), dns_overall and dns_p808 (deep noise suppression mos scores from p.835 and p.808 @xcite ), plcmos (packet loss concealment mos @xcite ), and ssqa (subjective speech quality assessment @xcite ). asr module specific: these metrics evaluate the system's ability to understand users' spoken utterances. in the absence of ground truth transcripts, we generate reference transcripts from 3 judge asr models: espnet librispeech asr @xcite , owsm 3.1 @xcite , and whisper @xcite . the asr transcripts generated by the cascaded system are then compared against these references to compute word error rate (wer) and character error rate (cer). text dialogue module specific: following prior work @xcite , we compute perplexity using gpt-2 @xcite and vert @xcite to evaluate the coherence and diversity of system outputs, respectively. additionally, the dialogpt @xcite model is used to compute the perplexity of system responses given the user utterance. to evaluate context modeling, we compute the similarity between the system response and the entire prior dialogue context using bert @xcite . conversation level specific: while the previous metrics focused on evaluating specific modules of the cascaded dialogue system and can be computed at each turn, we also support conversationlevel metrics (see sec. a.5 in appendix) to assess the system's ability to manage conversational flow. inspired by prior work @xcite , these metrics include turn-taking statistics, speaking rate (words per minute), and backchannel rate (number of backchannel words foot_5 per minute). e2e spoken dialogue models usually generate both text and audio tokens, enabling the computation of conversation-level metrics and some module-specific metrics. notably, tts-specific metrics and dialogue metrics like perplexity and vert can be computed without the need for the system to generate asr transcript. ## web interface fig. 1 shows a screenshot of an example gradio web demo built using our toolkit. the web demo collects the user audio in a streaming manner (fig. 1(a) ), with a vad module detecting when the user stops speaking. the processed audio is passed to the dialogue system. for cascaded systems, the output (fig. 1(b )) includes the system's audio response and 2 text boxes displaying the asr transcript and generated text response foot_6 . user audio is continuously collected to allow interruptions, enhancing interactivity. conversations are limited to 5 minutes, after which the interface refreshes. additional functionalities are discussed below. plug-and-play architecture: our web interface features a plug-and-play architecture (fig. 1(c )) that allows users to select from a variety of cascaded and e2e dialogue systems, enabling a total of 41 different system variations. details are provided in tab. 5 in the appendix. to avoid out-ofmemory errors, models are loaded on the fly, enabling seamless switching between systems. however, loading a new model may take some time, and the system output boxes are temporarily hidden during this process. on-the-fly evaluation: our demo supports realtime display of module-specific metrics (sec. 5) for each turn, including latency for each component. notably, user data is not recorded for on-the-fly evaluation; however, users may experience a brief wait for results. these metrics allow users to also gain quantitative insights into system performance, such as latency and audio quality. we encourage users to leverage these metrics during their interaction to pinpoint specific differences between dialogue systems, facilitating actionable insights. ## analysis using evaluation metrics to demonstrate the utility of our supported automatic evaluation metrics (sec.5), we use them to compare and analyze different dialogue systems and their submodules. this evaluation is conducted on the publicly available switchboard @xcite eval 2000 dataset, comprising 11 hours of human-human conversation data. full descriptions of the evaluation setup are in sec. a.2. ## conclusion and outlook to conclude, this work introduces espnet-sds, an open-source toolkit for creating user-friendly and interactive web interfaces for spoken dialogue systems. it provides the infrastructure to interact with a variety of cascaded and e2e spoken dialogue systems using a unified, gradio-based demonstration. the modular design of our codebase ensures easy extensibility to support more dialogue systems in the future. through initial evaluations, we demonstrate that the supported evaluation metrics offer valuable insights into the strengths and limitations of existing systems. we hope that this work inspires future research and serves as a useful resource for researchers aiming to compare and refine design choices in building spoken dialogue systems.