Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models (2025)

Abstract: Recent advancements in multi-turn voice interaction models have improved user-model communication. However, while closed-source models effectively retain and recall past utterances, whether open-source models share this ability remains unexplored. To fill this gap, we systematically evaluate how well open-source interaction models utilize past utterances using ContextDialog, a benchmark we proposed for this purpose. Our findings show that speech-based models have more difficulty than text-based ones, especially when recalling information conveyed in speech, and even with retrieval-augmented generation, models still struggle with questions about past utterances. These insights highlight key limitations in open-source models and suggest ways to improve memory retention and retrieval robustness.

## introduction voice assistants such as apple siri and amazon alexa have become an irreplaceable element of daily life, enabling natural and efficient speechbased interactions. in early systems, a cascaded pipeline is employed where speech is first transcribed using automatic speech recognition (asr), then processed as text, and finally converted back to speech via text-to-speech (tts) @xcite . with the advent of large language models (llms) @xcite , however, the research community has shifted greatly towards end-to-end approaches. these models integrate asr, text processing, and tts into a unified multimodal framework @xcite , which not only reduces latency (xie and wu, 2024a) but also better preserves the richness of vocal cues @xcite . in line with this trend, @xcite has demonstrated impressive capabilities by processing visual, speech, and text data in an end-to-end manner, where various voice interaction models, datasets, and benchmarks have rapidly emerged alongside. @xcite @xcite . despite these advances, most current models excel only in single-turn interactions. in practical applications, however, users engage in multiturn dialogs where a one-off response is insufficient. specifically, models must continuously retain and leverage contextual information from previous turns. for example, gemini 2.0 @xcite demonstrates the ability to remember preceding details-for instance when a user provides an apartment door code during interaction and inquires about it later-thereby showcasing robust context-maintenance. notably, other closed-source solutions, such as openai's advanced voice mode @xcite , have also showcased similar capabilities by referencing past interactions. in parallel, the open-source community has also intensified its efforts to develop voice interaction models that support multi-round communications @xcite @xcite . typically, these models take speech as input and generate both text and speech outputs, rather than producing spoken responses alone, to leverage the strengths of pre-trained llms and ensure coherent, multi-turn responses. however, it remains unclear whether current open-source systems can effectively retain and utilize long-range interaction histories. furthermore, there are no benchmarks that explicitly require leveraging dialog history to generate responses. in this work, we systematically investigate the ability of open-source voice interaction models to maintain and utilize conversational context through two key experiments. we evaluate (1) whether models can recall and generate spoken responses based on previous dialog and (2) their robustness in incorporating externally retrieved utterances. to support this evaluation, we introduce contextdialog-a speech-to-speech benchmark that focus on assessing recall via spoken question-answer (qa) pairs derived from existing spoken dialogs, prompting one speaker to reference earlier information. our findings reveal that open-source models struggle with past speech in two key aspects. performance gap with text-based systems -speech models generally perform worse than their textbased counterparts, and modality-based recall gap -within speech models, recalling speechbased information is less accurate than retrieving text, likely due to weaker speech processing capabilities. additionally, our investigation of retrievalaugmented generation @xcite shows that it fails to compensate for the model's inability to recall past information. we identify a major challenge: sensitivity to retrieval errors -models are highly susceptible to retrieval mistakes, leading to unchanged or even degraded performance. through these findings, we highlight the challenges models face in processing past conversational context and their sensitivity to noise in retrieved information, drawing attention to a fundamental, yet often overlooked, capability within the open-source community. our contributions are as follows: voice interaction models early voice interaction models follow a cascaded approach @xcite , transcribing speech into text, processing the transcription, and then synthesizing the output speech. recently, end-to-end pipelines have emerged, performing these steps within a single model @xcite . although some models generate spoken responses without relying on text @xcite , the inherent length and data scarcity of speech hinder semantic modeling @xcite . recent approaches integrate text generation within speech modeling to mitigate such problem, leveraging pre-trained llms by incorporating text as an intermediate representation @xcite , generating it alongside speech @xcite , or interleaving it with speech tokens @xcite . many prior works focus on single-turn voice interaction @xcite @xcite @xcite . as a natural extension, multi-turn voice interaction models have also emerged @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . among these, only slam-omni (chen et al., 2024c) and @xcite explicitly discuss long-context modeling. slam-omni seeks to improve multi-turn modeling by storing dialog history in transcribed text form, which is then prepended as a prefix during inference. lyra, on the other hand, explores and proposes various techniques to handle long audio histories and extends context windows directly within its model architecture. the remaining works either mention the use of multi-turn data for training or imply it through demonstrations or official implementations. however, they generally do not propose explicit methods or evaluations specifically targeting longcontext recall capability. consequently, whether these models can effectively handle past conversational history in real-world multi-turn dialog scenarios remains largely unexplored. for voice assistants to function effectively, it is essential to assess their ability to retain and utilize prior utterances to generate contextually appropriate responses. to this end, we select slam-omni (included in appendix a.1.6 due to its smaller model size) and lyra as our primary long-context baselines. additionally, we include freeze-omni, glm-4-voice, minicpm-o, and moshi (also in appendix a.1.6) as strong open-source candidates with multi-turn capabilities available. benchmarks numerous datasets and benchmarks for audio foundation models have emerged @xcite @xcite , particularly for voice interaction models @xcite @xcite et al., 2024; xie and wu, 2024a). for example, in task-oriented spoken dialogs, benchmarks assess a model's ability to recognize entities and dialog states from past utterances @xcite @xcite , while in open-domain dialogs, they focus on modeling and evaluating response coherence @xcite @xcite . beyond semantic relevance, recent works propose benchmarks targeting non-verbal components crucial for voice interaction models, such as gender, emotion, and background noise @xcite @xcite . unlike existing benchmarks that evaluate multi-turn semantics without ensuring past information is necessary for responses, contextdialog explicitly requires models to retrieve and utilize relevant past utterances, enabling a systematic assessment of recall ability. ## experiments in this section, we present the results of two experiments and analyses using contextdialog. in section 4.1, we demonstrate that open-source voice interaction models struggle to recall past information on their own, particularly user-specific information that exists solely in spoken form. then, in section 4.2, we show that even when leveraging a more advanced dedicated text retriever, models fail to respond robustly given the retrieved information, yielding limited improvements in spoken response generation. these two analyses highlight a critical yet often overlooked aspect of voice interaction models, their ability to remember past interactions, which is essential for real-world deployment. for our experiments, we select four open-source multi-turn voice interaction models: glm-4-voice @xcite , minicpm-o 2.6 @xcite , freeze-omni @xcite , and @xcite . to support real-time generation and minimize latency, these models generate responses directly from the input speech without an intermediate speech-to-text conversion. to mitigate semantic degradation in speech-only generation, these models generate text responses alongside spoken responses: glm-4-voice employs an interleaved token generation approach, alternating between text and speech tokens (figure 5(a) ), while minicpm-o, freeze-omni, and lyra generate text responses while simultaneously synthesizing speech using real-time generated text tokens and the llm's hidden states (figure 5(b) ). in all experiments, we evaluate each model's spoken response using the llm-as-a-judge approach @xcite , following previous works @xcite . this setting is denoted as @xmath0, s , where s refers to speech data and t to text data. the bold and underlined modality symbol indicates the evaluation target in each configuration. we employ gpt-4o-mini for evaluation, referred to as the gpt score in this paper, using a five-point scale where higher scores indicate better performance. we design prompts to assess recall by measuring how well the generated responses contain the ground truth information relevant to the given question, as detailed in appendix a.4. since gpt-4o-mini is tailored to text inputs, we first convert the spoken responses into text using whisper-large-v3 @xcite . additionally, considering that each model also generates an intermediate text response corresponding to the spoken output, we also evaluate it (@xmath1, s). by analyzing the evaluation results along with the word error rate (wer) between the text response and the transcribed spoken response, we can disentangle recall ability from speech synthesis capability, allowing us to identify cases where the model successfully recalls information but fails in speech synthesis. we use the official implementations, hyperparameters, and checkpoints for all four models (section a.2), running experiments on a single nvidia a40 gpu. detailed model descriptions and additional analyses are provided in appendix a.1. ## conclusion in this work, we conducted an in-depth analysis of a critical yet underexplored challenge in opensource voice interaction models: maintaining and utilizing past utterances. to address the lack of benchmarks that explicitly require accurate reference to past dialog, we introduced contextdialog, a speech-to-speech benchmark designed to systematically evaluate a model's ability to recall utterances from previous turns. using this benchmark, our experiments revealed that models struggle with recalling past utterances and remain highly sensitive to retrieval errors, limiting improvements even with dedicated retriever. these findings highlight a crucial gap in memory retention for open-source models, emphasizing the need for stronger conversational memory methods, such as improved long-context modeling, robust rag techniques, or dedicated memory modules. we hope that our work may act as a trigger to raise awareness to this overlooked challenge and encourage future research to further enhance the usability and effectiveness of voice interaction models.