URO -Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models (2025)

Abstract: Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose **URO-Bench**, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model’s abilities in **U**nderstanding, **R**easoning, and **O**ral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.

## introduction compared with traditional cascaded asr-llm-tts spoken dialogue systems, end-to-end speechto-speech (s2s) models like mini-omni @xcite and llama-omni (fang et al., 2025) † corresponding author. compared to text-based llms, the evaluation of sdms should consider both cognitive dimension including reasoning and knowledge, and speechrelated aspects such as speech quality and paralinguistic information. however, there is still a lack of comprehensive evaluation methods in s2s scenarios, which hinders us from understanding the real capabilities and shortcomings of current sdms, impeding the further development of spoken dialogue systems. besides, the transition to end-to-end modeling enables spoken dialogue systems to better comprehend and synthesize complex audio information like emotions, speaker characteristics, music, and ambient sounds. some commercial large speech language models, such as @xcite and @xcite , have already demonstrated such capabilities, while opensource frameworks @xcite @xcite still exhibit significant performance gaps. in this paper, we re-examine the process of the model engaging in speech interaction and introduce uro-bench, a comprehensive benchmark to assess s2s models' capabilities in understanding, reasoning, and oral conversation (figure 1 ). we selected questions suitable for speech dialogue scenarios from several widely used datasets, generated task-specific questions using @xcite , and synthesized the corresponding audio using state-of-the-art tts systems. our benchmark is made up of two different tracks: basic and pro, including 20 different s2s tasks. the basic track consists of ten english test sets and ten chinese test sets, which covers most of the application scenarios and tasks in real-time voice conversations, including life advice, common-sense qa, and calculations, with a goal of testing the model's general knowledge, instruction-following, and reasoning abilities. the pro track is composed of eleven english test sets, eight chinese test sets, and one multilingual test set. these highly challenging tests comprise multi-round dialogue, crosslingual conversation, and paralinguistics, assessing the sdm's capabilities of context management, linguistic adaptability, and non-verbal perception. this track is designed to incentivize advancements in future models, pushing the boundaries of performance on these challenging tasks. based on the comprehensive benchmark constructed, we evaluated 6 open-source sdms along with their backbone llms as reference. experiments reveal that current end-to-end spoken dialogue models perform relatively well on simple everyday conversation tasks and some models exhibit a foundational proficiency to manage multiround dialogues. however, most sdms still lag behind cascaded models (whisper + llms) and the state-of-the-art gpt-4o-audio-preview @xcite , with significant gaps in their instructionfollowing and reasoning capabilities. at the same time, most sdms demonstrate poor ability on multilingual tasks, and fail to handle situations related to paralinguistic information, highlighting the future direction for sdm development. all related code and datasets of uro-bench will be open-sourced, and a leaderboard to track the latest progress in this area will be maintained. we believe that uro-bench can effectively facilitate the development of spoken dialogue models with its comprehensive evaluation of current models and encouragement for future advances. ## conclusion in this work, we introduce uro-bench, aiming to provide a comprehensive benchmark for end-to-end spoken dialogue models. in particular, it is the first s2s benchmark that covers evaluations of multilingualism, multiround dialogues, and paralinguistics. extensive experiments and analysis are conducted on various open-source and closed-form sdms, revealing significant performance gaps in instructionfollowing and reasoning capabilities compared to cascaded models. furthermore, most sdms struggle with tasks related to multilingualism and paralinguistics, highlighting key areas for future development. we will open-source this benchmark and evaluation code, and maintain a leaderboard that provides a platform for the community to access and compare sdms performance over time.