Speech-Integrated Modeling for Behavioral Coding in Counseling (2025)

Abstract: Computational models of psychotherapy often ignore vocal cues by relying solely on text. To address this, we propose MISQ, a framework that integrates speech features directly into language models using a speech encoder and lightweight adapter. MISQ improves behavioral analysis in counseling conversations, achieving ~5% relative gains over text-only or indirect speech methods—underscoring the value of vocal signals like tone and prosody.

## introduction vocal cues such as tone, pitch, and prosody are crucial in counseling conversations, as they convey emotions and therapeutic intent @xcite . however, many current computational psychotherapy systems are based solely on text transcriptions, often derived from automatic speech recognition (asr) models. consequently, these models ignore important paralinguistic information @xcite , limiting their ability to fully capture the complexity of therapeutic interactions. the process of converting speech to text inherently leads to the loss of important vocal information @xcite since most asr systems prioritize linguistic content over non-verbal speech cues. as shown in figure 1 , even a correctly transcribed utterance (e.g. 0% word error rate (wer)) cannot fully capture the speaker's tone or emotional delivery, both of which are essential to understand the therapy-client interaction. recent work in automated mi analysis has predominantly relied on text transcriptions from asr systems, but growing evidence highlights the value of multimodal signals, such as acoustic cues and prosodic features, in capturing nuanced client behaviors @xcite @xcite . in this work, we seek to bridge this gap by exploring how speech features from pre-trained speech encoders can be leveraged to improve the modeling of counseling conversations. we focus on behavioral coding and forecasting for counseling conversations conducted using motivational interviewing (mi), a counseling strategy that facilitates behavior change by resolving ambivalence through reflective listening and collaborative dialogue @xcite . we introduce the multimodal integrated model for interactions in counseling (misq), to combine a speech encoder with a large language model (llm). we show that incorporating speech features significantly improves behavioral coding per-formance over text-only models, and that directly integrating raw speech consistently outperforms indirect methods, emphasizing its value in modeling counselor-client interactions. ## motivational interviewing with speech cues (misq) we build a multimodal language model, misq, by integrating speech information directly into our model using a speech encoder by processing text and speech features with the same underlying language model. this integration preserves the performance of llm in text-based tasks @xcite and allows the model to fully utilize the rich vocal information embedded in speech, such as tone and prosody, often lost in text-mediated approaches, such as paralinguistic captioning. ## results & analyses the results shown in in addition, we observe that the performance of the misq with empty sound inputs drops sharply. this significant decline emphasizes that for misq, speech information is crucial to understanding and predicting participant behavior, and text alone does not provide adequate information for behavioral modeling. however, in figure 3 , we also note that some confusion remains between misq's prediction of neutral, sustain talk, and change talk possibly due to inherent ambiguities in client speech. forecasting. in forecasting, misq again leads in performance (52.94% accuracy), surpassing the text baseline and paralinguistic captioning models. client's accuracy increased to 64.05%, and therapist's accuracy improved modestly. despite the inherent difficulty of forecasting, direct speech integration clearly provides helpful cues to anticipate conversational flow. robustness to noise. we also evaluate our model in challenging acoustic environments to assess whether it is capable of handling degraded audio signals. we simulate noisy acoustic environments by adding gaussian noise at varying signal-to-noise ratio (snr) levels to the raw speech input. our experiments show a strong robustness to the noise injected. as illustrated in figure 4 , misq shows minimal performance degradation in the categorization task at different noise levels. in contrast, the model experiences a more pronounced performance decline for the forecasting task in the presence of noise. this degradation is most severe at the highest noise level, whereas at lower levels, the impact on performance is relatively minor. we hypothesize that paralinguistic features in the speech signal may be more resistant to noise compared to purely linguistic and semantic cues. ## conclusion in this work, we presented a simple yet effective framework that integrates speech features to a pretrained text lm in order to improve the modeling of mi counseling sessions. by combining an llm with a lightweight adapter and a speech encoder, our speech integration approach, misq, achieves a general accuracy improvement of up to 5.5% over baselines of text alone (text baseline) and conditioned text (paralinguistic captioning), demonstrating its effectiveness in behavioral coding tasks. these results underscore the critical role of raw speech features in capturing the nuanced dynamics of counselor-client interactions. furthermore, our study highlights the importance of using direct speech integration for more accurate and insightful computational models in psychotherapy.