ALBA : Adaptive Language-Based Assessments for Mental Health (2024)

Abstract: Mental health issues differ widely among individuals, with varied signs and symptoms. Recently, language-based assessments haveshown promise in capturing this diversity, but they require a substantial sample of words per person for accuracy. This work introducesthe task of Adaptive Language-Based Assessment (ALBA), which involves adaptively ordering questions while also scoring an individual’s latent psychological trait using limited language responses to previous questions. To this end, we develop adaptive testing methods under two psychometric measurement theories: Classical Test Theory and Item Response Theory.We empirically evaluate ordering and scoring strategies, organizing into two new methods: a semi-supervised item response theory-basedmethod (ALIRT) and a supervised Actor-Critic model. While we found both methods to improve over non-adaptive baselines, We foundALIRT to be the most accurate and scalable, achieving the highest accuracy with fewer questions (e.g., Pearson r ≈ 0.93 after only 3 questions as compared to typically needing at least 7 questions). In general, adaptive language-based assessments of depression and anxiety were able to utilize a smaller sample of language without compromising validity or large computational costs.

## introduction standard mental health (e.g., depression, anxiety) assessment consists of asking patients a fixed set of questions to which they respond along a rating scale. for example, the patient health questionnaire (phq) asks, "over the last 2 weeks, how often have you been bothered by having little interest or pleasure in doing things? 0: not at all, 1: several days, 2: more than half the days, or 3: nearly every day" @xcite . this presents an information limitation: answering a fixed set of questions often leads to some unnecessary questions, while logging answers on a single dimensional rating scale limits the total information content possible. recent work has begun to address this information limitation by utilizing a patient's natural language to assess mental health conditions @xcite @xcite . such open-ended language responses enable participants to elaborate on their mental health, where appropriate computational methods can be used to quantify the language response to a semantic scale @xcite . patients find that language is more precise in communicating their mental health issues, preferring it to rating scales @xcite , but languagebased assessments can be lengthy @xcite often requiring word minimums @xcite . therefore, not only is there merit in mapping patients' language to their conditions but also in asking questions or prompting for language more optimally to reduce redundant information and adapt iteratively to each question. we introduce the task of adaptive language-based assessment (alba). alba is inspired by adaptive testing used in psychology and education @xcite , which usually utilizes a bayesian statistical framework known as item response theory (irt) over discrete-valued responses to questions. it helps adapt to the future prompts based on the prompts already administered to the participant, iteratively picking the next best question to ask based on the current latent estimate. we envision alba as a step towards the development of conversational diagnostic agents, as it allows for the modeling of language prompts in a semi-supervised manner. using this approach, agents can adaptively conduct language-based assessments with dynamic scoring and benefit from a prior understanding of responses, leading to improved diagnostic accuracy, less use of patient and clinician time, and more personalized interactions. our main contributions include: (1) introducing the task of adaptive language-based assessment (alba) for selecting questions that provide the most informative responses; (2) development of alirt model, an approach integrating predictive modeling to be able to apply irt to linguistic responses (as opposed to numeric responses as is typically used) -produces depression scores from only 4 questionresponses that have 90% of the variance explained of an assessment that uses 11 question-responses; foot_0 (3) development of an actor-critic model model for adaptive language-based assessments; (4) evaluation of different modeling strategies within these models (e.g., discretizing in 2-tomous); (5) extensive empirical comparison of these methods with more straight-forward approaches covering multiple scoring strategies, and (6) insights into the questions that generally produce responses with the most information (i.e. ability to distinguish participant depression severity) informing better question/prompt creation. ## background item response theory estimates a latent variable as a proxy for an unobservable attribute (such as depression) by modeling the interaction between: (1) a latent variable (e.g. depression "score"), (2) observable variables from a population of participants (e.g. responses to questions foot_1 about lack of sleep, appetite, etc), and (3) the data points from a particular individual who is to be assessed. the latent score is typically estimated with bayes parameter estimation. irt addresses a short-coming of classical test theory (ctt) @xcite which is based on the assumption that there is a true score for an unobservable attribute, which is typically taken to be the summed value of all observable variables (e.g., ratings), and that the observed estimate is only off of the true score by some error from the act of measurement. this ignores correlations between the observable variables. in this work, we compare scores based on both irt and ctt while evaluating the proposed adaptive testing methods over standard questionnaires for measuring major depression (phq-9) and generalized anxiety disorder (gad-7). 3 various irt modeling functions have been proposed for capturing discrete responses @xcite @xcite . increased model complexity in irt leads to a rise in parameters, requiring larger datasets. given the multidimensional nature of language representation and limited advances in simultaneous modeling across dimensions, we employ polytomous item response theory @xcite to discretize language responses onto a graded scale. in this work, we introduce adaptive languagebased irt (alirt), which uses a supervised approach to polytomize the linguistic responses, and then employs adaptive testing using irt. ## methods the components that we develop for alba aims at (1) dynamically ordering a set of questions, picking next-best at a time; and (2) scoring the assessment at each step. these components can either be modeled jointly (as in alirt) or step-wise (as in our actor-critic model). we describe both approaches below and a suite of more straightforward baseline approaches to compare against. ## dataset our dataset consists of open-ended language answers to eleven questions and two self-diagnostic tests in the form of closed-ended rating scales. participants were recruited online from mechanical open-ended questions shorthand word-response correlation with the phq-9 describe how you generally felt the last 2 weeks, that is, how you felt on average. describe mental health 0.61 describe how you have been feeling about yourself over the last 2 weeks. describe yourself 0.61 over the last 2 weeks, have you been depressed or not? describe depression or not 0.58 over the last 2 weeks, have you been worried or not? describe worry or not 0.41 overall in your life, are you in harmony or not? describe harmony or not 0.54 overall in your life, are you satisfied or not? describe satisfaction or not 0.57 describe the nature of your physical movements over the last 2 weeks (have you for example been moving and speaking slowly; or the opposite, been fidgety and restless). ## results & discussion we report the performances of the various adaptive strategies, in comparison with the baselines and across scoring methods, in ## related work over the past decade, researchers have been exploring techniques for mental health assessment @xcite . initial studies inspired by leveraging communication in social media, indicated that nlp models could moderately accurately predict self-disclosed mental health conditions or events @xcite , scores from self-report mental health questionnaires @xcite , and achieve scores aligned with standard screening surveys when compared to clinical records of depression @xcite . however, such methods only work well with a fairly active social media usage @xcite . while some have proposed methods to utilize transformers with smaller datasets @xcite , such an approach is still limited to those willing to share such data or having any of it at all. further, it has recently been shown that the accuracy of language-based assessments can reach even greater (@xmath0.8) when the assessment is based on prompting participants for language responses related to mental health, mirroring standard questionnaires but using language responses instead @xcite . still, past work has mostly been validated against summed, or averaged, questionnaire scores, while here we consider improved measurement paradigms that rely on latent variables, such as item-response theory @xcite . within the domains of nlp, irt has been used in chatbot evaluation @xcite , for textual entailment @xcite . irt has also been used to impute missing data @xcite and to compare different ml classifiers at an instance level. feature/question selection (an np-complete problem) has also been explored with irt over a number of fixed selection, ranking, and ordering methods in the recent years @xcite @xcite . in a related study, (coban, 2022a) applied irt to linguistic data, converting language into a term-document matrix for feature selection. however, our approach in adaptive language-based assessment extends beyond fixed feature selection settings. we aim to dynamically adapt to each data sample, facilitated by irt-based ordering. it's important to note that adaptive language testing differs from personalized recommendation systems. while the latter emphasizes item similarity, language-based testing strives to precisely assess users' latent traits, setting it apart from recommendation systems designed for preferences. ## conclusion mental health issues vary widely across individuals, suggesting the need for assessments that can enable wide ranging symptoms and be adaptive to the individual. we introduced the task of adaptive language-based assessment for eliciting the most informative responses as well as developed and explored two methods to perform the task, alirt and the actor-critic methods, along with a suite of more straight-forward approaches. evaluated against depression severity scores derived from 11 questions, alirt was able to capture over 90% of the variance explained (r 2 ) after only 4 questions while optimal fixed ordering approaches needed at least 7, suggesting patient time could be saved with this approach. we further saw that a regression approach that tries to optimally weight questionscores had only minor benefits over the irt-based ( l), that alirt generalized to assessing anxiety in addition to depression, and that symptom-focused questions were not as informative (never chosen early) as compared to broader questions. the adaptive approach, in general, can significantly reduce the number of questions required to achieve high validity, as well as yield insights into the questions that produce the most informative responses suggesting better question/prompt creation.