V ox RAG : A Step Toward Transcription-Free RAG Systems in Spoken Question Answering (2025)

Abstract: We introduce VoxRAG, a modular speech-to-speech retrieval-augmented generation system that bypasses transcription to retrieve semantically relevant audio segments directly from spoken queries. VoxRAG employs silence-aware segmentation, speaker diarization, CLAP audio embeddings, and FAISS retrieval using L2-normalized cosine similarity. We construct a 50-query test set recorded as spoken input by a native English speaker. Retrieval quality was evaluated using LLM-as-a-judge annotations. For very relevant segments, cosine similarity achieved a Recall@10 of 0.34. For somewhat relevant segments, Recall@10 rose to 0.60 and nDCG@10 to 0.27, highlighting strong topical alignment. Answer quality was judged on a 0–2 scale across relevance, accuracy, completeness, and precision, with mean scores of 0.84, 0.58, 0.56, and 0.46 respectively. While precision and retrieval quality remain key limitations, VoxRAG shows that transcription-free speech-to-speech retrieval is feasible in RAG systems.

## introduction traditional question-answering (qa) retrievalaugmented generation (rag) systems retrieve text documents from a vector database by performing semantic similarity search from a user's query. a large language model (llm) then generates context-aware answers based on the retrieved content @xcite . this architecture, however, can be extended to operate directly on spoken audio instead of text. retrieving spoken audio documents without relying on intermediate transcriptions is an emerging area of rag research @xcite . we present voxrag: a modular, open-source retrieval pipeline for rag with full speech-to-speech retrieval. unlike hybrid text and audio systems, voxrag keeps both the user query and retrievable documents in audio form up to the genera-tion stage, using contrastive language-audio pretraining (clap) embeddings @xcite to retrieve semantically relevant segments directly from podcast audio (see appendix d for sample qa pairs). using podcasts as a retrieval target presents challenges such as informal language, overlapping speakers, non-speech audio (e.g., music, laughter), and generally poor automatic speech recognition (asr) transcription output quality @xcite . voxrag mitigates these issues with silence-aware segmentation, speaker diarization, and clap embedding retrieval, avoiding early commitment to potentially faulty transcripts and enabling semantically grounded retrieval in the acoustic domain. we evaluate both retrieval and answer quality using ragelo's llm-as-a-judge methods, which have shown positive alignment with human judgments in qa evaluation, to assess how well retrieved audio supports answer generation @xcite . related work has explored rag systems for audio in both text and hybrid modalities. the trec 2020-21 podcasts track saw systems using asr text retrieval and summarization @xcite , including fine-tuned bart @xcite and whisper spoken term detection. hybrid systems like schwertzer's combination of cola @xcite and roberta @xcite show promise in mixed-modal retrieval @xcite . more recent models embed audio and text into shared or comparable vector spaces. speechdpr distills from asr and dense passage retrieval (dpr) systems to embed spoken passages directly @xcite , while seal uses separate encoders for speech and text to enable cross-modal retrieval without transcription @xcite . spectron processes spectrograms for qa entirely within an llm framework @xcite , and speechrag integrates speech retrieval with an llm for answering text queries from raw audio @xcite . meanwhile, dual demonstrates fully speech-native retrieval by embedding discrete speech units without paired text training @xcite . voxrag contributes to this emerging space by exploring a retrieval-first, speech-native architecture that maintains audio representations up to the point of answer generation, differing from span-prediction models like dual. ## method our construction of voxrag was motivated by two core research questions: 1) can we retrieve semantically relevant documents directly from spoken language and without relying on text representations? 2) can those documents support high-quality answer generation using an llm? we define "high-quality" segments as those that contain very relevant or somewhat relevant information to a user's query. we define "high-quality" answers along four axes: relevance, accuracy, completeness, and precision. ## discussion our results highlight the challenges inherent in speech-to-speech retrieval within unstructured, multi-speaker podcast content. although clap embeddings provided a degree of coarse semantic alignment, the retrieval process often prioritized topically related segments over precise matches. this tendency resulted in lower precision and incomplete responses. evaluations revealed strong correlations between accuracy, completeness, and precision, indicating a shared reliance on finegrained factual grounding. in contrast, relevance scores remained consistently high, suggesting that topical alignment alone is insufficient for generating high-quality answers. certain queries, particularly those involving lifestyle concepts such as "shower," achieved perfect scores in some cases, but not reliably. this inconsistency may reflect variability in how well specific topics are represented within audio embeddings and warrants further investigation. the modular architecture of voxrag enabled rapid experimentation across embedding models, chunking strategies, and retrieval logic. the inclusion of audio playback within the interface proved valuable for error analysis, as it revealed retrieval mismatches that were not apparent from text alone. these findings establish a baseline for future research on audio-native question answering. they point to the need for improved embedding finetuning, more effective segmentation methods, and reranking strategies that better reflect factual precision. voxrag represents a step toward multimodal rag systems capable of operating directly on realworld, noisy, and informal spoken content. with the proliferation of audio media, systems of this kind will be increasingly important for enabling direct retrieval and reasoning over speech, without dependence on textual transcripts. ## conclusion voxrag explores the viability of a fully speech-tospeech retrieval pipeline for retrieval-augmented generation. while the system ultimately produces text answers, it retrieves documents directly from audio using clap embeddings @xcite , bypassing early transcription. despite its novel architecture, the system underperforms on precision, completeness, and accuracy metrics, highlighting the limitations of current audio embedding models for fine-grained semantic retrieval. however, the retrieval quality on certain queries demonstrates the ultimate viability of rag with speech-to-speech retrieval.