Generating Mental Health Transcripts with SAPE ( S panish Adaptive Prompt Engineering) (2024)

Abstract: Large language models have become valuable tools for data augmentation in scenarios with limited data availability, as they can generate synthetic data resembling real-world data. However, their generative performance depends on the quality of the prompt used to instruct the model. Prompt engineering that relies on hand-crafted strategies or requires domain experts to adjust the prompt often yields suboptimal results. In this paper we present SAPE, a Spanish Adaptive Prompt Engineering method utilizing genetic algorithms for prompt generation and selection. Our evaluation of SAPE focuses on a generative task that involves the creation of Spanish therapy transcripts, a type of data that is challenging to collect due to the fact that it typically includes protected health information. Through human evaluations conducted by mental health professionals, our results show that SAPE produces Spanish counselling transcripts that more closely resemble authentic therapy transcripts compared to other prompt engineering techniques that are based on Reflexion and Chain-of-Thought.

## introduction large language models (llms) have achieved state-of-the-art (sota) performances on multiple natural language processing (nlp) benchmarks @xcite . they have the potential to be adapted as data augmentation tools in scenarios where data is hard to collect @xcite . however, sota llms such as gpt-4 and bard are only accessible as services, withholding access to the model's underlying architecture and parameters for commercial reasons @xcite . consequently, fine-tuning these models for specific downstream nlp tasks becomes infeasible. as an alternative to enhance llms without the need to retrain them, prompt engineering has proven to be an effective approach @xcite . different prompt techniques have shown a significant impact on a model's arithmetic problemsolving capabilities @xcite , enhance the model's ability to accurately mimic human writing styles @xcite , or improve the model's commonsense reasoning skills @xcite . however, these prompt engineering techniques require manual crafting. given that the precise phrasing of a prompt can significantly influence its effectiveness @xcite , studies have been directed towards automating prompt engineering @xcite . it is noteworthy that the exploration of prompt engineering has predominantly centered on the english language, leaving other languages largely unexplored in this domain. therefore, this paper introduces sape, a spanish adaptive prompt engineering method that employs genetic algorithms for prompt generation and selection. to evaluate the generative capabilities of llms using sape's prompts, we employed openai's text-davinci-003 model to create synthetic therapy transcripts. psychologists compared sapegenerated transcripts with those produced using prompts based on chain-of-thought (cot) @xcite and reflexion @xcite methods. our results indicate that overall mental health professionals find sape-generated text to resemble authentic therapy transcripts more closely than texts generated with other prompt engineering techniques. we chose to focus on generating synthetic psychotherapy data since obtaining authentically generated therapy transcripts is challenging due to privacy concerns and the need to protect sensitive health information @xcite . the development of high-quality synthetic ther-apy transcripts holds great potential for advancing nlp models in mental health. these transcripts can train text classifiers to automate post-therapy session reports, thus streamlining administrative tasks for psychologists. furthermore, they can be utilized to train chatbots that simulate mental health patients, which could provide (trainee) psychologists or counsellors with a valuable tool for training and practicing their skills, without the need for other interlocutors. in addition to the applications of nlp, collections of psychotherapy transcripts offer a valuable resource for study in psychology and counselling. for instance, the counseling and psychotherapy transcripts: volume i collection from alexander street press (alexander street press, 2023) provides a substantial set of therapy transcripts suitable for teaching and research purposes. however, these transcripts are predominantly in english, lacking equivalents in other languages. the sape tool, and similar adaptations in other languages, presents an opportunity to create sets of therapy transcripts in non-english languages where authentic collections are scarce. our main contributions are as follows: ## results in this section, we present our statistical findings derived from the evaluation tasks completed by mental health professionals. the subsequent section is dedicated to a comprehensive discussion and analysis of the implications arising from these outcomes. ## discussion the only therapeutic interaction prompt in which sape did not get the highest ranking was for the "set goals" prompt. a notable characteristic distinguishing this prompt is the inclusion of a preamble explicitly indicating it as an instruction, unlike the other prompts that solely specify the actions expected from the model (consult appendix a for sape's prompts). when we use text-davinci-003 to hypermutate a mutation prompt or use a mutation prompt to evolve a prompt, we specify that the new prompt should be given without any preamble. the deviation in the "set goals" prompt raises a compelling point for discussion. despite the capability of larger models to handle more intricate tasks @xcite , we made a deliberate decision not to employ the latest and most expensive models from openai due to budget constraints, acknowledging the potential decrease in performance of our automatic prompt engineering technique. however, it is pertinent to explore whether the deviation for the "set goals" prompt also emanated from prompting the model in spanish rather than english. further research is imperative to comprehensively compare task performance between different languages, particularly for automatic prompt engineering techniques reliant on language model feedback for refinement. if a model underperforms in a task for a specific language, self-referential automatic prompt engineering techniques may be less effective. in such cases alternative prompt engineering techniques should be adapted to account for such limitations. for example, in our scenario, the instruction entailed both evolving and formatting a prompt. an alternative approach could have involved separate calls to the llm: one to mutate the prompt and another to format it. however, this approach might not always be feasible when using models behind paid apis due to associated additional costs. another point of discussion pertains to the evaluation of the final population's fitness. notably, for all three prompts, sexual reproduction emerged as the mutation prompt generating the lowest fitness. future work can explore changing or removing the sexual reproduction mutation prompt to enhance sape's performance in comparison to the cot and reflexion based approaches. however, it is also essential to address the limitations of the creation process of the cot and the reflexion prompts. despite being crafted by domain experts in the mental health field, domain expertise alone does not ensure an optimal prompt design. the ability to refine and modify a prompt to elicit pertinent responses from llms is a distinct type of skill. while it is valuable to compare different prompt engineering techniques against each other, future work could focus on comparing synthetic versus organic data. while sape shows promising results, additional work is required to test if its quality is comparable to real therapy transcripts. the ongoing improvement of llms and the refinement of prompt engineering techniques contribute to the consistent improvement in the quality of synthetic transcripts. the generation of synthetic transcripts comparable to organic ones holds significant benefits to the mental health research community. this advancement could serve as a valuable data augmentation technique, alleviating challenges associated with accessing authentic data. ## conclusion we introduced sape, a spanish adaptive prompt engineering technique that employs genetic algorithms for the evolution and selection of prompts. to assess sape's prompt quality, we conducted a comparative analysis between the text generated using sape-derived prompts and text generated from prompts based on cot and reflexion techniques. a series of statistical tests revealed a statistically significant preference for the text generated by sape for the cumulative scores rankings of the generated synthetic texts. the type of data we generated using sape pertains to psychotherapy transcripts, a type of data that is challenging to collect due to its sensitive nature. furthermore, the data was produced in spanish, adding to its significance, as acquiring substantial datasets in languages other than english poses additional difficulty. while sape was designed for generating spanish synthetic data, we consider that the evolutonary algorithm it employs can be extended to other languages and used for creating diverse synthetic datasets beyond mental health data. we hope that our algorithm will be adapted for various languages and fields, thereby facilitating the creation of datasets that would oth-while synthetic therapy transcripts offer significant potential as a tool for data augmentation in training nlp models within the mental health domain, they also raise notable ethical considerations. in our experimental approach, sape focused on discovering prompts that yield therapy transcripts closely mirroring authentic ones. however, its search strategy does not explicitly consider or assess potential biases introduced by the chosen prompt. llms can manifest various types of biases in their outputs, necessitating an examination to prevent the inadvertent propagation of such biases @xcite @xcite . in the context of synthetic mental health data, it is crucial to assess the presence of any stereotypes in the texts. studies have shown that stereotypes and biases exert adverse effects on mental health treatment outcomes @xcite . given that llms are trained on vast volumes of data (e.g., gpt-3 trained on 45tb of text data), there exists the potential risk of including private information in the training data @xcite . hence, it is crucial to ensure that synthetic transcripts do not inadvertently disclose real individuals' identities or personal information. the process must strictly adhere to data protection laws, such as gdpr in europe or hipaa in the united states, to safeguard personal data. moreover, transparency in both the creation and utilization of synthetic transcripts is imperative. researchers must clearly outline the methods employed to generate synthetic data, along with acknowledging the inherent limitations of such transcripts. there exists a risk that certain synthetic therapy transcripts may not accurately capture genuine therapeutic interactions, potentially resulting in misconceptions or misinterpretations of mental health conditions and therapeutic practices. thus, it is crucial to validate synthetic transcripts against real-world data and involve psychotherapy experts in reviewing and refining the synthetic generation process, thereby enhancing accuracy and reliability.