Conversational Topic Recommendation in Counseling and Psychotherapy with Decision Transformer and Large Language Models (2024)

Abstract: Given the increasing demand for mental health assistance, artificial intelligence (AI), particularly large language models (LLMs), may be valuable for integration into automated clinical support systems. In this work, we leverage a decision transformer architecture for topic recommendation in counseling conversations between patients and mental health professionals. The architecture is utilized for offline reinforcement learning, and we extract states (dialogue turn embeddings), actions (conversation topics), and rewards (scores measuring the alignment between patient and therapist) from previous turns within a conversation to train a decision transformer model. We demonstrate an improvement over baseline reinforcement learning methods, and propose a novel system of utilizing our model’s output as synthetic labels for fine-tuning a large language model for the same task. Although our implementation based on LLaMA-2 7B has mixed results, future work can undoubtedly build on the design.

## introduction in recent years, there has been a notable uptick in the number of people seeking professional help for mental health concerns, but the available pool of mental health professionals remains small in comparison. to address this need, automated ai-based tools and methods for counseling have been explored and engineered, ranging from systems for training junior mental health counselors @xcite to ai-in-the-loop chatbots @xcite . with the dramatic rise in popularity and accessibility of large language models (llms), it's expected that llms will play a significant role in the intersection of computing and mental health research, as well. in our prior work @xcite , we introduced the supervisorbot, a reinforcement learning (rl)-based topic recommendation system in counseling conversations. this proves to be a useful tool for clinicians during their psychotherapy sessions, where the system recommends what topics to discuss next given what has been discussed so far, as well as what works best in the past in terms of the patient outcomes. in this work, we improve upon this meaningful task by introducing the decision transformer @xcite , a transformer model designed for reinforcement learning (rl), into the recommendation pipeline, demonstrating better performance than other rl methods. we also explore the potential combination of decision transformer with llms, by generating labels for unseen transcript data using the pre-trained decision transformer model, and feeding the synthetically annotated data to fine-tuning a llm. our primary contribution is demonstrating that in the task of topic recommendation, decision transformer outperforms baseline rl methods; if such a system were to go through the process of user testing, the decision transformer--or models building on or improving decision transformer--can be utilized as the backbone for the recommendation module. we first describe how we implement the preprocessing of the therapy conversation dataset, and how this is fed into the decision transformer model. we then describe how we use a portion of the dataset to train the decision transformer model, and that trained model's predicted labels are used as input to a large language model to train for the same task of topic recommendation. ## related work decision transformer was introduced as a transformer-based architecture to abstract the process of offline reinforcement learning, and has been used successfully in various nlp tasks including natural language understanding @xcite , navigating text-based games @xcite , and generative language modeling @xcite . the decision transformer architecture has also been effectively applied to the clinical domain to generate treatment recommendations based on patient history @xcite . in this work, we effectively apply the decision transformer architecture to the mental health domain in a dialogue recommendation task and improve on performances with older reinforcement learning methods. the improvement of ai-in-the-loop tools to support humans in tasks has typically focused on human feedback, although more recent work has explored the potential for ai tools to improve themselves through a number of methods. @xcite demonstrates that a generative language model can improve its own outputs through fine-tuning on its own generations, and that the improvements are more significant as the model size increases. generative models also have the advantage of being able to generate improvements to their own outputs @xcite . in addition to models improving themselves, ensemble methods in which one model serves some intermediary purpose within the pipeline--e.g. data generation or filtering for input to another model--can be used for conversational modeling tasks as well @xcite . @xcite ) uses an intermediary model's output as a reward function for another model, outperforming sole supervised learning from the source dataset. in this work, we explore a potential pipeline in which one model's output is used as synthetic data to train a language model for the task of topic recommendation. we consider the idea of ai supplementing a typical reinforcement learning with human feedback (rlhf) process by experimenting with how ai may be able to augment feedback, which can have significant implications given the lack of publicly available mental health dialogue data, let alone annotated data. ## architecture in the following sections, we describe the architecture of our system in detail (see fig. 1 ). ## limitations due to limitations of computational resources, experimentation with fine-tuning llms is restricted by model size. future work can build on this work by applying similar experiments on increasing model sizes or non-quantized versions of models, effectively demonstrating (positively or negatively) that performance scales with model size. ## ethical considerations when implementing a topic recommendation system in counseling contexts, ethical considerations are important due to the sensitive nature of digital mental health discussions, as discussed in @xcite . one of the primary concerns is the potential limitation imposed by a static set of discussion topics. while such a system can streamline the counseling process, it risks limiting the creativity and flexibility of counselors, particularly those in training, and in the long term, inhibit consideration of their own perspectives on how to continue the conversation. this could inadvertently restrict their ability to tailor sessions according to the unique needs of each patient. this is particularly relevant since the topics pulled are from one specific dataset that covers only four mental health conditions. the training dataset, derived from this limited number of mental health conditions, might not be representative of the broader population or other conditions. this limitation can lead to biased recommendations if not carefully managed. to mitigate this, it is essential to consider a more dynamic approach where the set of topics can evolve based on ongoing input from practicing counselors and feedback from therapy sessions. this adaptation would help in maintaining the relevance and sensitivity of the recommendations to diverse patient needs. in deployment, we can also imagine that topics are dynamically chosen, or chosen using human feedback; for example, perhaps before the system is put into use, counselors can input their own topics. in addition to dataset limitations, the calculation of rewards, based on the working alliance inventory (wai), while rooted in established psychological theory, may benefit from enhancements through reinforcement learning with human feedback (rlhf). incorporating direct input from users could refine the understanding and alignment of counselor and patient goals, improving the system's effectiveness and ethical alignment. ## conclusion in this study, we introduced a decision-transformer-based recommendation system which outperforms baseline rl-based methods in counseling topic recommendation, indicating that transformer-based methods may have better performance in general when it comes to modeling conversation direction and alignment. we additionally find that the model performs best for certain reward scales on shorter input sequences, indicating that some exploration of optimal sequence length can be an avenue for future work. through additional analysis of the attention scores, we additionally find that the model pays more attention to items earlier on in the input sequence.