Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation (2025)

Abstract: End-to-end speech-to-speech (S2S) dialogue systems have recently garnered increasing research attention for their lower latency and more natural integration of nonverbal cues such as emotion and speaker identity. However, these systems face key challenges, particularly in incorporating external knowledge, a capability commonly addressed by Retrieval-Augmented Generation (RAG) in text-based large language models (LLMs). The core difficulty lies in the modality gap between input speech and retrieved textual knowledge, which hinders effective integration of information. To address this issue, we propose a novel end-to-end RAG framework that directly retrieves relevant textual knowledge from speech queries. Experimental results demonstrate that our method significantly improves the performance of end-to-end S2S dialogue systems while achieving higher retrieval efficiency. Although the overall performance still lags behind the SOTA cascaded models, our framework offers a promising direction for enhancing knowledge integration in end-to-end S2S systems. Our code and dataset are released.

## introduction the release of gpt-4o @xcite , a unified model capable of comprehending and generating text, speech, and visual modalities within a single neural architecture, represents a significant milestone in the development of multimodal large language models (mllms). mllms are defined as large language model-based architectures that can perceive @xcite , reason over @xcite , and generate @xcite multimodal information @xcite , with representative models including anygpt @xcite and qwen-omni @xcite . among various mllm paradigms, endto-end speech-to-speech (s2s) dialogue models, such as mini-omni @xcite , llama-omni @xcite , and slam-omni @xcite , constitute an important subclass, aiming to directly respond to speech input with speech output through a single integrated framework. however, due to the nature of the modality gap between input speech and retrieved textual knowledge, these models lack the capability to retrieve external knowledge documents and face limitations in timely knowledge updates and domainspecific information acquisition, hindering their performance in question-answering tasks. retrieval-augmented generation (rag) @xcite , which retrieves relevant information from an external source and incorporates it as contextual input to enhance response accuracy, effectively addresses the limitation of text-based llms in accessing external knowledge @xcite . however, in end-to-end speech-to-speech dialogue models, directly retrieving textual documents is often ineffective due to the difficulty of aligning input speech with the corresponding textual knowledge. a straightforward solution is to leverage automatic speech recognition (asr) to convert speech into text and then use the recognized text for retrieval. while this approach can be effective, high-accuracy asr models, such as whisper @xcite , introduce additional latency, which poses a critical limitation for real-time vertical spoken assistants. therefore, balancing retrieval accuracy and latency remains a critical challenge in building highperformance voice-based retrieval assistants. given that rag relies on encoding textual information into a vector space for indexing and retrieval, we propose an end-to-end approach that embeds speech queries and textual content into a shared representation space, enabling direct speech-to-text retrieval without needing asr. this approach significantly reduces retrieval latency and streamlines the pipeline. in our experiments with speech-to-speech dialogue models, the proposed end-to-end rag approach reduces retrieval latency to one-fourth of that incurred by the asr-based cascaded rag models. specifically, on spoken questionanswering tasks, our model achieves an effective improvement in answer accuracy over the baseline without retrieval augmentation (eg, 20% improvement on hotpotqa @xcite ), although performing lower than the cascade models employing asr-based cascade retrieval. these findings highlight the effectiveness of our approach and establish a promising design paradigm for future research on multimodal rag. in summary, our key contributions are as follows: ## conclusion in this work, we highlight the efficiency bottleneck in cross-modal retrieval, spark discussion around the trade-off between latency and accuracy, and contribute an open-sourced, multilingual end-toend retrieval augmented generation framework to facilitate further research. the proposed method leverages a pretrained embedding model with multiple encoders, allowing both speech and text to be indexed within the same vector space, which enables direct retrieval of relevant textual information from speech queries. the experimental results demonstrate that our method significantly enhances retrieval speed, which is an efficient and scalable framework for end-to-end speech-to-speech dialogue systems. furthermore, we point out that the key factor in improving retrieval accuracy is enhancing the semantic alignment of the shared vector space, which provides guidance for future research.