When LLM s Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection (2024)

Abstract: Depression is a critical concern in global mental health, prompting extensive research into AI-based detection methods. Among various AI technologies, Large Language Models (LLMs) stand out for their versatility in healthcare applications. However, the application of LLMs in the identification and analysis of depressive states remains relatively unexplored, presenting an intriguing avenue for future research. In this paper, we present an innovative approach to employ an LLM in the realm of depression detection, integrating acoustic speech information into the LLM framework for this specific application. We investigate an efficient method for automatic depression detection by integrating speech signals into LLMs utilizing Acoustic Landmarks. This approach is not only valuable for the detection of depression but also represents a new perspective in enhancing the ability of LLMs to comprehend and process speech signals. By incorporating acoustic landmarks, which are specific to the pronunciation of spoken words, our method adds critical dimensions to text transcripts. This integration also provides insights into the unique speech patterns of individuals, revealing the potential mental states of individuals. By encoding acoustic landmarks information into LLMs, evaluations of the proposed approach on the DAIC-WOZ dataset reveal state-of-the-art results when compared with existing Audio-Text baselines.

## introduction depression, a common mental disorder affecting 10-15% of the global population, is characterized by persistent low mood, loss of interest, and lack of energy, making it a prevalent and costly illness @xcite . given the timeconsuming, expensive, and sometimes ineffective nature of traditional depression treatment methods, a growing number of researchers are turning their attention to developing automated depression detec-figure 1 : example of acoustic landmark (2-gram concat landmark (g+p-), (s+p+), (p+,p-), ..., (g-b-)), landmarks are extracted from abrupt changes in the speech signal. they can discretize speech into a series of tokens that possess linguistic significance. tion systems. concurrently, large language models (llms) have recently demonstrated remarkable success across a variety of tasks @xcite . these large language models have been applied to various healthcare issues, including general surgery @xcite , dementia diagnosis @xcite , and gastroenterology @xcite and achieved excellent results. however, their main limitation stems from their sole reliance on textual input, which limits their full potential. simultaneously, the use of large language models (llms) in depression detection remains largely unexplored. in particular, there has been no effort to integrate speech-despite growing evidence that speech signals can reveal indicators of depression @xcite )-into these llms, an advancement that could greatly improve their effectiveness in identifying depression @xcite . one of the key approaches to incorporating speech signals into llms is through the discretization of speech. however, the current landscape of speech discretization, heavily reliant on deep learning techniques @xcite , faces significant challenges due to its considerable gpu memory requirements. this is particularly problematic in the field of depression detection, where data often consists of lengthy conversations @xcite . the need for completed conversations is vital for accurate depression detection @xcite , rendering the existing deep learning-based methods impractical for such applications. for this purpose, it is necessary to find an efficient approach that allows for the discretization of speech with reduced gpu memory usage. acoustic landmarks represent event markers intricately linked with the articulation of speech, forming a concise alternative framework for speech processing @xcite . this approach emphasizes the analysis of abrupt acoustic changes at the subsegmental level, thereby providing a succinct and precise phonetic description of language. these landmarks, characterized by their binary values, establish a minimal yet effective set for differentiating each language segment from others. they maintain a direct and significant relationship with acoustic properties and articulation (including individual pronunciation), ensuring discernibility despite unwanted variability introduced by diverse hardware and environmental backgrounds @xcite . their discrete nature not only allows for efficient integration into large language models but also offers a viable alternative for understanding speech signals in depression detection, bypassing the limitations of current deep learning-based techniques. this innovative approach promises a more feasible and resource-efficient pathway for analyzing complex speech patterns in mental health diagnostics. in this paper, we introduce a novel multimodal approach to depression detection, utilizing a combination of acoustic landmarks and large language models. we investigate the properties of large language models at various stages and under dif-ferent conditions after integrating landmark-based speech information. we investigate how llms learn speech landmarks and assess the impact of conversational fine-tuning on the performance of llms in tasks related to depression detection. in summary, our contributions include the following: ## ablation study and discussion in this chapter, we conduct an empirical study to meticulously analyze and elucidate the characteristics of llms that we identified in the context of depression detection during our experiments. ## conclusion this paper introduces an efficient approach for depression detection using acoustic landmarks and llms. this approach is not only valuable for the detection of depression but also represents a new perspective in enhancing the ability of llms to comprehend speech signals. furthermore, we are the first to research multimodal depression detection using llms. we establish a new benchmark with a sota f1-score of 0.84 through ensemble learning. additionally, we evaluated various peft methods and discovered that applying lora across all layers yields identical outcomes for both p-tuning and lora in depression detection. our analysis further reveals how llms process speech landmarks, guiding future research in this domain.