Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization (2024)

Abstract: In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs’ ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue’s structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.

## introduction mental health counseling serves as a crucial frontline defense against mental illness. in a typical counseling session, clients articulate their issues while therapists provide support. an essential component of these sessions involves building a strong therapeutic bond and documenting the entire dialogue, commonly known as counseling note, figure 1 : the proposed pipeline allows llms to first plan and then generate. in our approach, prioritizing planning before generation enriches summarization with conversational structure and domain knowledge. while recent advancements in large language models (llms) for mental health are enormous @xcite , it is crucial to incorporate fundamental domain knowledge and understanding of the structural nuances of the dialogue, which current research lacks in ability @xcite . consequently, we require a knowledge planner to adeptly capture domain intricacies and determine generation priorities. for instance, figure 1 illustrates a sample counseling conversation between a therapist (t) and a client (c). unlike conventional approaches that directly generate summaries using foundational models, our proposed pipeline introduces a planner. through this planner, we infuse nuanced dialogue structure and domain-specific knowledge into llms. earlier efforts in counseling summarization @xcite focused on utilizing domain knowledge by integrating patient health questionnaires (phq) and counseling components. however, it fell short compared to recent llms. the broader scope of such models has been explored for incorporating knowledge into generated text @xcite and controlling text generation @xcite . while these efforts are focused on general-purpose downstream tasks, the complexity of counseling dialogue necessitates introducing a more (a) domain-centric and (b) knowledgestructured approach to cater to this problem. our study introduces a novel planning engine, designed specifically to guide the generation of llms. focused on enhancing counseling summarization, we employ mentalllama as the foundation to develop our framework -piece, planning engine for mental counseling note generation. our model's planning engine exploits knowledge scaffolding and sheaf learner by integrating domainspecific and structural knowledge into llms. we evaluate our piece against several llms, such as mistral, zephyr, and llama. to assess the effectiveness of our approach, we compare piece's performance against 14 baseline methods. our quantitative evaluation consists of four automatic summary evaluation metrics -rouge: r-1, r-2, r-l, and bluert, along with a domain-centric metric, mental health information capture (mhic) proposed by @xcite . we observe a clear improvement of 3.42%, 10.11%, and 6.01% across r-1, r-2, and r-l metrics, respectively. furthermore, to assess the generalizability of piece, we experiment on aci-bench, a clinical note generation dataset @xcite . we observe that piece surpasses the state-of-the-art llms with planning-engine on aci-bench as well. additionally, we perform an extensive expert evaluation through an established clinical relevance framework on a set of six dedicated metrics and three task-relevant survey questionnaires. the results affirm the superiority of piece against the baselines and demonstrate the adaptability of the planning engine across various llms. our contributions are summarized below: • we propose piece that integrates the planning engine with mentalllama to address the issue of unreliable generation by llms. this engine plans the llm's generation by filtering dialogue and injecting domain and structural knowledge. ## related work we present our literature review under three major segments to understand and build a planner for llm-based counseling summarization. ## dataset we utilize memo, a common counseling summarization dataset with 191 counseling dialogues with 11, 543 utterances @xcite . each dyadic counseling dialogue contains a conversation between a therapist (5, 722 utterances) and a client (5, 814 utterances), along with an expert annotated counseling summary. in addition to this, memo further contains each utterance labeled with a counseling component -symptom and history (sh), patient discovery (pd), reflecting (rt), or discus-sion filler (df) (c.f. appendix d for definitions). the dataset contains 2379 utterances tagged with sh, 5428 utterances tagged with pd, and 1242 utterances tagged with rt. these counseling components act as labels of relevance for utterances. ## proposed methodology our proposed model, piece, operates on the memo dataset and aims to generate knowledgeenriched counseling summaries for a counseling dialogue containing n utterances, @xmath0, u 2 , u 3 , . . . u n ⟩. there are two specific kinds of knowledge that piece focuses on -(a) (mental health) domain knowledge and (b) structural understanding of dialogue. the model achieves this knowledge integration through a novel planning engine that filters and fuses relevant information to guide the generation of knowledge-enriched summaries by the underlying llm. figure 2 presents a schematic diagram of piece. ## experiments and results here, we discuss the selection of baseline methods followed by the performance comparison with baselines, the ablation of piece, and analyses. ## conclusion in our work, we explored the research space of mental health counseling summarization. our research focused on strategically orchestrating llms to plan before generating content, thereby incorporating relevant dimensions of conversational structureand domain-centric knowledge into the summarization process. we proposed a novel planning engine for llms, integrated and presented in this research with mentalllama as piece. we compared the performance of piece against a spectrum of 14 potential baseline methods, including state-of-theart llms like mistral, llama-2, flan-t5, zephyr, and mentalllama. next, we presented an extensive evaluation spanning automatic, human, and expert evaluations. expert analysis indicated that the summaries generated by piece exhibit better relevance and structure, often surpassing the closest competitive llms and, in certain instances, even surpassing the gold standard. we conclude by discussing our research's ethical considerations and applicability, emphasizing our approach as an assistive module tailored exclusively for mental health experts, which mitigates potential risks associated with direct client interactions, ensuring a safeguard against unintended harm. this strategic positioning underscores our commitment to safe utilization in mental health contexts.