LLM Questionnaire Completion for Automatic Psychiatric Assessment (2024)

Abstract: We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The LLM is prompted to answer these questionnaires by impersonating the interviewee. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C), using a Random Forest regressor. Our approach is shown to enhance diagnostic accuracy compared to multiple baselines. It thus establishes a novel framework for interpreting unstructured psychological interviews, bridging the gap between narrative-driven and data-driven approaches for mental health assessment.

## introduction psychiatric evaluation, nowadays, is heavily based on patient verbal reports of disturbed feelings, thoughts, behaviors, and their changes over time. it consists of two main parts: unstructured interviews, where patients express themselves freely with openended questions, and structured questionnaires to standardize the assessment. these methods, outlined in the diagnostic and statistical manual of mental disorders (dsm), aim to assign universal scores to individual experiences (american psychiatric association, 2013). however, the complexity of mental health, characterized by a positive manifold of symptoms and the subjective, often unreliable nature of self-reports (especially between sessions), as well as interviewer bias, makes accurate diagnosis difficult @xcite . the overlap of symptoms and unstable mental states, particularly in pathological conditions, adds to the challenge of achieving precision, limiting an objective quantitative account of subjective self-experience @xcite @xcite . the evolution of psychiatric practice is increasingly shaped by integrating natural language processing (nlp) and machine learning into traditional diagnostics @xcite . this shift moves from a theory-driven to a data-driven paradigm, addressing the limitations of conventional psychiatric assessments. nlp plays a key role in overcoming the subjective interpretation of unstructured interviews and the rigidity of standardized questionnaires. by using large-scale text data and advanced language models, nlp adds nuanced, patient-specific insights to psychiatric evaluations. this data-driven approach complements existing theories, fostering a more precise and holistic understanding of mental health, aligning with the precision medicine trend in healthcare. we propose a two-step method for psychiatric evaluation using unstructured interview text. first, the llm is asked to complete multiple questionnaires while impersonating the interviewee. these include (i) established psychiatric questionnaires (phq-8, pcl-c), and (ii) custom questionnaires created with gpt-4, covering mental health, personality traits, and therapeutic aspects. in the second step, the llm's responses are used as features for a random forest regressor @xcite , which predicts the interviewee's scores on the two clinical questionnaires. ## related work earlier diagnostic efforts in mental health @xcite @xcite @xcite incorporate both textual and speech intonation features, recognizing the role of tonal elements in depression assessment. @xcite manually extract six psychological dimensions (sleep quality, ptsd/depression diagnosis, treatment history, introversion, personal preference, and feelings) from interview text, then apply a random forest regressor. recently, transformer architectures have driven significant progress in depression diagnosis. @xcite use roberta @xcite to encode interview segments, focusing on symptom prediction, while zhang and guo (2024) introduce prompt learning and an attention mechanism for enhanced accuracy. these approaches rely on supervised learning, whereas our work utilizes zero-shot impersonation. galatzer-levy et al. ( 2023 ) explored the med-palm 2 llm @xcite for psychiatric prediction, not by answering questionnaires but by predicting outcomes using prompts like, "based on the following clinical interview, what do you estimate the participant's [phq-8/pcl-c] score is?" concurrently, @xcite applies promptbased few-shot learning @xcite to analyze social media content, answering questions like "what mental disorder symptoms does this post show?" with a focus on textual explanation quality. ## method this work leverages the extended distress analysis interview corpus (e-daic) @xcite , a dataset of semi-clinical interviews designed to assess psychological distress. to build the corpus, an autonomous ai interviewer was employed to minimize human bias and two clinical questionnairebased scales, phq-8 and pcl-c, were used to measure both depression and post-traumatic stress disorder (ptsd). our analyses adhere to the predefined training, development, and test splits as specified in the dataset documentation. the e-daic dataset comprises @xmath0, each associated with two psychiatric scores quantifying the severity of depression and ptsd, measured using the phq-8 and pcl-c standardized scales, respectively. given a text in the domain of psychological transcripts t @xmath1, where @xmath2, 2, . . . , 275, our objective is to design a feature extractor f : @xmath3. this extractor converts a given transcript t i into a d-dimensional feature vector v i such that the two ground truth scores our method, language model for impersonation-based questionnaire completion (lmiq), defines a feature extractor f as the transformation of unstructured interview transcripts into a structured array of impersonated responses to a list of questions. for each subject and question, the method: (1) generates a query by combining the transcript with a psychological questionnaire, (2) processes the query through an llm to obtain responses, and (3) stores the responses in an array, which represents the feature vector for that subject. the system prompt directed the llm to interpret transcripts and simulate responses as the interview subject. the exact prompt used in our research is detailed in appendix a.1, and a shorter reference prompt is as follows. attached are a psychological interview transcript and psychological questionnaire. analyze the conversation transcript and capture psychological insights about the speaker. deduct answers to the accompanied questions as if you were the speaker. the answer should be a numerical value ranging from 1-5, with 1 depicting "don't agree at all" and 5 depicting "very much agree." conversation transcript: {transcript} psychological questionnaire: {question} in our implementation, lmiq utilizes gpt-3.5 turbo instruct, accessed via the openai api ( https://platform.openai.com/docs/models/gpt3-5 ), for transcript analysis and predicting responses to psychological questionnaires. additionally, we evaluated lmiq using mixtral 7bx8 @xcite as its llm backbone. both llms were employed with their default configurations, including temperature settings, to ensure consistent and unbiased processing. the questions from the questionnaires were submitted to the llm one at a time in separate sessions to prevent bias from question ordering. the list of questions used by lmiq includes the original eight-item phq-8 and seventeen-item pcl-c questionnaires, as well as several additional questionnaires we developed using openai's chatgpt-4. we prompted it to create multiple fiveitem sets across three domains: (i) clinical mental health (dsm-5 guided), (ii) the five-factor personality model, and (iii) therapeutic-relevant domains. the broader range of questionnaires, beyond phq-figure 1 : lmiq pipeline overview. a. top. the extended-daic dataset, comprising 275 psychological interviews with associated ptsd and depression scores derived from the pcl-c and phq assessments. bottom. 5-item questionnaires across multiple mental health and personality domains. b. main prompt logic. for each instance, a task description is merged with a psychological interview transcript and a questionnaire, generating 5 impersonated responses. c. feature vector formation: aggregate the 5 simulated responses from each participant to construct a feature vector of dimension @xmath4, which is then used to train a random forest model for accurate prediction of the original assessment scores. 8 and pcl-c, addresses psychiatric comorbidity and symptom overlap. within each domain, gpt-4 was prompted to define multiple topics and within each topic to generate exactly five questions, each aiming to quantify symptoms on a scale between 1 and 5. these initial outputs were manually refined to ensure higher relevance and precision. the obtained domains are listed in tab. 1, the full list of questions is provided in appendix b; as a sample, the generalized anxiety disorder questions include: (1) do you find it hard to control your worrying? (2) does your anxiety interfere with your work, school, or family responsibilities? the feature vector was formed by concatenating responses (each an integer between 1 and 5) to five questions across twenty-four topics, resulting in a vector with @xmath5. this vector was then used as input for a random forest regressor implemented with scikit-learn @xcite . model optimization involved a hyper-parameter search, specifically examining 'n-estimators': @xcite @xcite and 'max-depth': @xcite @xcite . the configuration yielding the best ## experiments we employ multiple relevant baselines and various ablated versions of our method to demonstrate the advantage of our approach in predicting psychiatric scores from psychological transcripts. unless otherwise specified, all feature extraction methods are paired with the same random forest regression used for lmiq. ## results and discussion additionally, lmiq, using the mixtral 7bx8 model-which is believed to outperform gpt-3.5-shows improved results, highlighting the effectiveness of our approach and the potential of newer llms. the next best method predicts only clinical questionnaires, summing the results instead of using a regression model. this simplified lmiq approach, while still employing impersonation, performs only slightly worse than the full method. a full set of experiments examining the contribution of each questionnaire type is detailed in appendix c. the zero-shot method, which directly predicts questionnaire scores, consistently underperforms. the classic tf-idf encoding method yields the highest mses in both tasks. gpt-3.5 analyze & embed and mentallama 7b analyze & embed show mixed results but consistently outperform the method that directly embeds the interview with ada-002. ## conclusions lmiq leverages an llm as an impersonator to complete both established and llm-generated clinical questionnaires based on a subject's unstructured psychological dialogue, feeding these into a classifier for diagnosis. although we do not validate the impersonated responses due to the lack of available answers, we demonstrate their effectiveness in predicting phq and pcl scores. despite llms' limitations in numerical accuracy in some contexts, our results confirm the meaningfulness of the assigned values, both directly and through regression trees.