S peech T - RAG : Reliable Depression Detection in LLM s with Retrieval-Augmented Generation Using Speech Timing Information (2025)

Abstract: Large Language Models (LLMs) have been increasingly adopted for health-related tasks, yet their performance in depression detection remains limited when relying solely on text input. While Retrieval-Augmented Generation (RAG) typically enhances LLM capabilities, our experiments indicate that traditional text-based RAG systems struggle to significantly improve depression detection accuracy. This challenge stems partly from the rich depression-relevant information encoded in acoustic speech patterns — information that current text-only approaches fail to capture effectively. To address this limitation, we conduct a systematic analysis of temporal speech patterns, comparing healthy individuals with those experiencing depression. Based on our findings, we introduce Speech Timing-based Retrieval-Augmented Generation, SpeechT-RAG, a novel system that leverages speech timing features for both accurate depression detection and reliable confidence estimation. This integrated approach not only outperforms traditional text-based RAG systems in detection accuracy but also enhances uncertainty quantification through a confidence scoring mechanism that naturally extends from the same temporal features. Our unified framework achieves comparable results to fine-tuned LLMs without additional training while simultaneously addressing the fundamental requirements for both accuracy and trustworthiness in mental health assessment

## introduction large language models (llms) @xcite have been extensively utilized in health-related applications, achieving notable success in tasks such as medical evidence summarization @xcite , supporting decision-making in general surgery @xcite , and assisting in gastroenterological diagnoses @xcite . these models have demonstrated exceptional capabilities in transforming traditional healthcare workflows by efficiently processing complex medical data and enhancing decision-making processes. however, in specialized domains like depression detection, llms based solely on text have shown limited effectiveness @xcite @xcite , often requiring extensive taskspecific fine-tuning and large annotated datasets to achieve satisfactory results. this reliance on significant training resources makes their application to depression detection both time-consuming and costly. when the performance of llms is restricted, retrieval-augmented generation (rag) is often employed as an alternative. by allowing llms to retrieve task-specific information from external sources during inference, rag has shown promise in mitigating the need for extensive fine-tuning across many tasks @xcite . however, in the context of depression detection, the limited performance of rag highlights the inherent constraints of text-based approaches shown in our experiment. these limitations stem from text's inability to capture critical non-verbal cues, such as tone, prosody, and timing, which play a significant role in identifying depression @xcite . therefore, it is crucial to address these gaps by seeking additional sources of information that can complement text-based methods and overcome their inherent limitations. numerous studies have demonstrated that acoustic features in speech, particularly temporal and prosodic patterns, contain rich information highly relevant to depression detection, and many existing works have successfully leveraged these acoustic characteristics as the primary modality for this task @xcite . however, directly integrating acoustic features into textbased llms has often been detrimental to their performance @xcite , making multimodal rag a promising alternative. in computer vision, rag systems frequently utilize salient visual elements or regions of interest (such as objects or scene segments) as input to retrieve relevant information @xcite , but an analogous technique for identifying and utilizing key acoustic segments is lacking in the speech domain. furthermore, speech-enabled llms typically rely on encoders capable of processing only up to 30 seconds of audio @xcite @xcite , which limits their applicability for tasks requiring longterm context @xcite @xcite . this constraint poses a significant challenge for depression detection, as the task often necessitates analyzing extended acoustic patterns across entire conversations @xcite . given these limitations, directly using speech llms is impractical for this domain, prompting the need to explore alternative approaches that leverage the advantages of textbased llms while effectively incorporating acoustic information from speech. ## speecht-rag framework our methodology involves three steps: first, we extract acoustic landmarks and construct sequential bigram pairs, embedding temporal information through their durations. second, we identify statistically significant landmark pairs that exhibit notable differences between the two groups based on their duration statistics. finally, we leverage these timing keys and text-based representations of temporal information to perform rag with llms. ## confidence score estimation confidence estimation is a critical component of health-related ai systems, offering a measure of reliability for predictions, which is essential for clinical decision-making @xcite . while traditional machine learning systems frequently employ confidence scoring techniques @xcite , their application to llms remains challenging @xcite . to address this, we propose a framework based on gaussian process classifiers (gpcs) that leverages speech timing information, specifically the temporal features of landmark bigrams with p-values below 0.05, to predict confidence scores. gaussian process classifier using speech timing information. the gpc models the relationship between input features and class labels within a probabilistic framework. here, the input feature vector @xmath0.g., mean, variance) of durations from landmark bigrams identified as significant (@xmath1.05) in the earlier analysis, where d is the dimensionality of the feature vector. the class label @xmath2, 1} denotes either the depression or health category. the gpc predicts class probabilities p (y | x) using a radial basis function (rbf) kernel: expected calibration error (ece) ece is a widely used metric to evaluate the quality of confidence scores by measuring their alignment with observed accuracies @xcite . for n test samples, ece is computed as: ## conclusion in this paper, we have introduced speecht-rag, a novel retrieval-augmented generation framework that leverages acoustic temporal patterns for depression detection and uncertainty assessment. through systematic analysis of llm attention mechanisms and speech characteristics, we demonstrated how temporal information embedded in acoustic landmark pairs can simultaneously serve two critical functions: capturing depression-related patterns (such as reduced energy levels and diminished vocal expressiveness) and providing a natural basis for assessing prediction reliability. the dual utility of these temporal features enables our unified framework to not only achieve strong detection performance but also offer interpretable confidence estimation without additional computational overhead. these findings demonstrate how domainspecific temporal patterns can enhance both the accuracy and reliability of llm frameworks, advancing the development of trustworthy systems.