Identifying Early Maladaptive Schemas from Mental Health Question Texts (2023)

Abstract: In Psychotherapy, maladaptive schemas– negative perceptions that an individual has of the self, others, or the world that endure despite objective reality–often lead to resistance to treatments and relapse of mental health issues such as depression, anxiety, panic attacks etc. Identification of early maladaptive schemas (EMS) is thus a crucial step during Schema Therapy-based counseling sessions, where patients go through a detailed and lengthy EMS questionnaire. However, such an approach is not practical in ‘offline’ counseling scenarios, such as community QA forums which are gaining popularity for people seeking mental health support. In this paper, we investigate both LLM (Large Language Models) and non-LLM approaches for identifying EMS labels using resources from Schema Therapy. Our evaluation indicates that recent LLMs can be effective for identifying EMS but their predictions lack explainability and are too sensitive to precise ‘prompts’. Both LLM and non-LLM methods are unable to reliably address the null cases, i.e. cases with no EMS labels. However, we posit that the two approaches show complementary properties and together, they can be used to further devise techniques for EMS identification.

## introduction background: psychotherapy researchers and practitioners have noted that patients with maladaptive schemas 1 do not respond fully or resist traditional cognitive-behavioral treatments and tend to relapse @xcite . schema therapy (st), introduced by @xcite , addresses this issue and builds on the widely-employed cognitive behavioral therapy @xcite and other counseling theories to conceptualize and treat mental health conditions that are due to personality disorders. st has seen §authors contributed equally. 1 defn. 3 at https://dictionary.apa.org/schema increasing adoption in recent counseling practice due to its effectiveness @xcite @xcite . post: am i going to be alone forever? i feel like i'm trying to convince myself that i'm okay when i'm not. i'm always blocking out the bad things and forgetting. i also feel like nobody cares for me and they never will. i feel truly alone. ## methods schema therapy-based counseling sessions often include the administration of a lengthy and detailed "young's schema questionaire" (ysq) as an initial step. this questionaire comprises 232 statements about oneself that are rated on a scale of 1 ("completely untrue of myself")-6 ("describes me perfectly") by the counselee. 5 for example, two statements from ysq pertaining to the ems labels for the question in the counselor, who is familiar with the association of ysq statements with the 18 ems labels, is able to assess the potential maladaptive schemas, if any, through the scores the counselee provides in the questionaire and further confirmatory questions. the form-filling exercise described above is not practical in the 'offline' scenario of community qa forums, requiring us to mimic this process indirectly with automated methods. towards this goal, we harness nlp associations between the ysq items and sentences from forum questions to infer how the ysq form could be probably filled by the counselee. let {q 1 . . . q m } refer to sentences in the input question q, @xmath0, . . . y n } represent the statements from ysq, and label(y ## results and observations dataset: for experimental evaluation, we consider the dataset from counselchat.com that was curated by @xcite . this representative dataset contains about 3000 questions seeking assistance and responses from licensed therapists for various mental health topics. from the publicly-released anonymized dataset which was carefully de-identified and curated from the website by bertagnolli (2020), 6 we considered the subset of 310 questions on the topics: anxiety, depression, trauma, self-esteem and anger-management where schema therapy is applicable. a random sample of about ten percent from this collection was chosen for expert annotation. annotation quality: to ensure the ethics, quality, and reliability of the annotations for this specialized task, we followed previous works @xcite and selected two counselors with verified credentials and self-disclosed practical experience in schema therapy on upwork. 7 the two counselors assigned ems labels with justifications for the question texts for about five examples per hour and were paid between 40 -50 usd an hour, as requested. between the two annotators, we obtained 30 questions each annotated with up to three most applicable ems labels, on average. about 17 examples were annotated by both independently, and on this subset the fleiss kappa (1971) value is 0.412 indicating moderate agreement. recent psychology-related studies report kappa values indicating a wide range of agreement from low to medium depending on the complexity of the factor that is measured @xcite . the moderate agreement value for our dataset sets the benchmark for ems prediction since the annotations were made by professionallyqualified counselors who practise schema therapy. we also note however, the possible effect of our annotation guidelines that ask for the top-3 labels for each post. considering the above aspects, we used the union of labels as the "ground truth" for evaluating our models. further details on the annotation process as well as the label distribution for our dataset are included in appendix a.4. ## related work community qa (cqa) forums are attracting significant nlp research due to their ubiquitious usage in the recent age of information sharing @xcite @xcite . recent research works on mental health topics address symptom extraction, triage characterization as well as identification of specific conditions such as depression and self-harm behaviour @xcite @xcite @xcite @xcite . unlike these previous works, we target the identification of underlying causes behind the manifested symptoms by characterizing them using ems labels from schema therapy. we posit that such a theoretical grounding using domain-specific frameworks is essential for reliably interpreting model predictions in mental health applications @xcite . ## conclusions in this paper, we proposed the novel task of identifying early maladaptive schemas in mental health question texts from community qa forums. to this end, we studied prediction techniques using both llm and non-llm approaches based on schema therapy, examined their strengths and weaknesses for the task, and identified their possible complementary nature regarding performance and explanations. for future, we would like to investigate if few-shot and exemplar-based approaches @xcite @xcite can overcome the drawbacks of llms highlighted in our study. we would also like to investigate recent research on incorporating human feedback through reinforcement learning @xcite and contrastive learning to further improve prediction performance @xcite .