The ethical role of computational linguistics in digital psychological formulation and suicide prevention. (2022)

Abstract: Formulation is central to clinical practice. Formulation has a factor weighing, pattern recognition and explanatory hypothesis modelling focus. Formulation attempts to make sense of why a person presents in a certain state at a certain time and context, and how that state may be best managed to enhance mental health, safety and optimal change. Inherent to the clinical need for formulation is an appreciation of the complexities, uncertainty and limits of applying theoretical concepts and symptom, diagnostic and risk categories to human experience; or attaching meaning or weight to any particular factor in an individual?s history or mental state without considering the broader biopsychosocial and cultural context. With specific reference to suicide prevention, this paper considers the need and potential for the computer linguistic community to be both cognisant of and ethically contribute to the clinical formulation process.

## introduction modelling is central to mental healthcare. deficits in modelling, or failure to understand and manage those deficits, can lead to deficits in care. risk prediction, the diagnostic process, and key phenomena identification and monitoring such as mood symptoms are valid targets for the application of computational linguistics to suicide prevention. however, from a clinical perspective each of these targets and the research and categorical conceptual modelling that underlie them has major limitations, complexity, and contention @xcite @xcite @xcite . many aspects of mental health clinical practice are based on limited theoretical models, limited data and limited resources and involve varying presentations, preferences and levels of understanding, strengths, insight, and engagement. formulation is the key clinical process for attempting to integrate these multiple interacting limited models and factors, to create an overall working model on which to base future action and interventions ( @xcite @xcite . clinical formulation has a pattern recognition, factor weighing and explanatory hypothesis modelling focus. formulation attempts to make sense of why a person presents in a certain state at a certain time and context and how given the known vulnerabilities, strengths, preferences and available resources that state may be best changed in a safe and effective way @xcite @xcite @xcite . in keeping with the evolution and variation of mental health practice, formulation has historically taken varying forms and had varying drivers, and had questions raised about its validity and utility. however formulation retains a central role in care delivery, is considered as requiring the highest level of clinical expertise and is a key component of examination for specialist qualification ( @xcite @xcite . inherent to the clinical need for formulation is an appreciation of the complexities, uncertainty and limits of applying categories and theoretical concepts to human experience or attempting to attach meaning or weight to any particular factor in an individual's history or mental state without considering the broader context. this paper considers the opportunities and challenges for computational linguistics in emulating and augmenting the clinical formulation process and contributing to broader related digital mental health developments. highlighted is the need to appreciate the ethical and clinical safety risks, particularly if developments in the computational linguistics field are misperceived or exaggerated in terms of their certainty and capacity for suicide prediction and reduction. the paper discusses the clinical assessment and planning process and the phenomenological psychopathology analysis, nosological diagnostic classification, individual psychodynamics and risk prediction complexities that drive the need for formulation. the concepts of mood, affect and emotion are discussed to illustrate some of the issues around the standardised interpretation of human experience and classification into diagnoses. the ethics and difficulties of attempting to predict or modify the risk of low base rate complex emergent events such as suicide is highlighted @xcite . a structure for formulation is provided to highlight the key components and where computational linguistics may be of assistance. the central arguments will be that the data gathering, pattern recognition, factor weighing, and modelling of clinical formulation are areas in which computational linguistics could and should assist. pattern recognition and modelling around words and language in context is central to mental health clinical practice and computational linguistics. mental health and computational linguistics specialists can synergically use language as a method to gain insight and formulate a model of another's consciousness, intent and experience. this can contribute to risk, diagnostic and psychodynamic formulation. however, appreciation of the limitations of modelling and prediction particularly in application to suicide prevention will remain central. the artificial intelligence (ai) ethical principles of autonomy, justice, beneficence non-maleficence and explicability will remain a challenge and a duty for the clpsych community @xcite . appreciating the rationale for the utilisation of formulation in clinical practice and seeking to place the ethos and process of formulation at the heart of computational linguistics practice to enhance explicability will assist in addressing that duty. machine learning and computational linguistics may play a role in more accurately identifying the contextual and contingent factors and the level of certainty or uncertainty inherent in the formulation modelling and explanatory hypothesis. the primary purpose of this work is to provoke thought and facilitate further conceptual and operational ethical co-design of digital formulation. the aim is to help build a shared understanding of the rationale, structure and process of clinical formulation and call upon the clpsych community to consider what contributions they could make to digitally enable and improve it particularly within a suicide prevention context. it is recognized there is a concept-reality gap between what clinicians might ideally desire and what the computational linguistic field is currently able to offer @xcite . however, the clpsych community could play an important role in clarifying and developing the conceptual vision for digital formulation, and the required technological and methodological steps to get there. the paper is intentionally largely technology and data source agnostic and focused on the clinical need and related medicolegal and ethical principles. the aim is to stimulate rather than limit thought or argue for a particular technological or methodological direction. the paper will touch on the initial steps to cross the concept-reality gap the authors are taking. this includes a focus on ethics, digital transformation, sleep and suicide, social media data and integrated thematic analysis and topic modelling. ## the role and place of formulation in clinical practice this next section aims to briefly set out some key concepts on which to build a shared understanding of the need for and place of formulation in clinical practice particularly in suicide prevention. these concepts are complex and contentious with differing definitions and scopes and varying degrees of clinical understanding and application in practice. highlighted are the roles and limitations of language, phenomenology, nosology and risk prediction. ## language as a window into mental and brain state there is limited understanding of the nature of consciousness or the mind and how this relates to brain function @xcite . however, there is a general understanding that integrated biological, psychological and sociological factors impact on brain function and impact on the integrated experience and expression of thoughts, emotion, and behaviour. machine learning affords the capacity to dynamically identify and analyse multiple signals indicative of an individual's mental state and intent. these signals may be neurophys-iological, behavioural and of increasing interest to suicide prevention natural language, including that occurring in social media @xcite @xcite . to gain a greater timely understanding of the lived experience and meaning of suicidal thoughts and behaviour we need a greater appreciation of the dynamic cognitions and emotion and contexts that colour an individual's thoughts and drive them to action @xcite @xcite . social media data may provide an additional window and insights into this experience and an opportunity to intervene in a timely way. clinically language is a key tool for assessing and communicating thoughts, emotion and behaviour. language is central to the assessment of mental state and from this potential brain state. language assists in making hypotheses about electrochemical and cognitive processes in a section or circuit of the brain at a particular point in time that hence drive physical, biological, psychopharmacological and psychosocial interventions. language is a significant window into human experience but may not always provide an accurately drawn picture of reality. the image may be skewed and distorted by faulty mental models, cognitive biases, and misinterpretations by both the experiencer and the observer. computational linguistics as the study of language using computational methods and theoretical models, similarly to clinical practice, has an inherent interest in ensuring any model deficits or conflicts are understood, minimised and managed. ## phenomenological psychopathology and nosology phenomenological psychopathological analysis is the process that underlies the clinical perception and interpretation of the experience and behaviour of others @xcite @xcite . nosology is the classification of medical diseases. nosological modelling can occur at three levels: aetiological (disease cause is known) pathogenetic (disease process is known) and symptom (only reported or interpreted experience is known). mental disorder diagnoses are typically at the symptom modelling syndrome level @xcite . human experience and behaviour are characterised by a dimensional nature and multifactorial temporal contextual determinants. complexity, and ambiguity is inherent. there is only limited knowledge of the causes and mechanisms by which mental disorders and perceived aberrant experience and behaviour arise. accordingly, there are only theoretical models of varying fidelity and evidence base and agreement around the nature and classification of mental disorder, and how experience and behaviour should be interpreted and determined to be pathological. similarly, the selection and mechanism of action of interventions, their benefits and harms, and predictions and determinants of prognosis all require the interpretation and weighing of various population research models as to what may be best and available for a specific patient in a specific mental state, in a specific time and context. risk categories, and diagnostic categories based on the identification and interpretation of phenomena and syndromes have significant reliability, validity and intervention, prognostic and safety limitations @xcite . although the terms affect, emotion and mood are often used interchangeably, they have a broad historical range of interrelated but separate specific meanings, definitions and perceived implications arising from variations @xcite . an emotion can be understood as the subjective personal experience and interpretation of a feeling state. affect refers to an assessor's interpretation of the emotional experience of another, and typically includes not just reference to the type, but also the range and stability and appropriateness of expressed emotion within a specific context. emotions may be of short duration and fluctuate and represent the subjective interpretation of chemically induced physiological experience. the interpretation of this physiological emotional experience may be influenced by the longer standing and more prominent mood state, which may have a complex biopsychosocial basis. a report of a mood symptom such as depression may have significant differing impact, relevance and meaning depending on the pattern intensity, duration, associations and context of occurrence. it may be a sign of a brief adjustment to a stressor, an indication of emotional dysregulation in someone with a personality disorder, form part of various levels and presentations of a major depressive disorder, be associated with medical and neurological disorders from dementia to parkinson's disease, be associated with or secondary to drug use prescribed and illicit, form part of a broader bipolar disorder, or be an early presentation or association with schizophrenia. weight may be given to one diagnosis over another if there is a clear family or personal history or pattern of a particular disorder and other known risk, symptom, sign and contextual factors are present. diagnoses can be of use in care planning, funding, research and making predictions about the future. however, they have significant limitations, not least if it is forgotten they are syndromal level models, that tell little of the personal story and context of the individual. the symptom and sign and temporal components of the diagnostic model may be subject to deficits or non-standardisation in interpretation and report. race, culture, gender, age, language, education, intellectual and sensory impairment and economic status and societal marginalisation may all have an impact on the expression and interpretation of experience and behaviour. these factors may contribute to significant inter-rater variability as to what diagnosis or diagnoses are ascribed to an individual. individuals that receive a specific mental disorder diagnosis, may have significant variation in terms of what criteria they meet, their individual experience, and underlying causal and mechanism of development factors. for example, in the dsm classification system 227 combinations of criteria can lead to a diagnosis of major depression, including 64 combinations which don't require a report of depressed mood. some combinations may be more common and more meaningful from a clinical priority and potential to intervene perspective and computational linguistics could assist with identifying these @xcite . ## suicide risk prevention there are major challenges, limitations and clinical and ethical risks in trying to predict complex multi-factorial emergent low base rate events such as suicide that have a high magnitude of adverse consequence if that prediction is wrong @xcite @xcite . the majority of those classified as being at high risk of suicide, do not commit suicide, and the majority of suicides will emerge from those classified as low risk, or who have not been assessed for suicide risk, have not expressed suicidal ideation or whom are not engaged in services @xcite @xcite . the expression and actioning of suicidal intent can vary in intensity and fluctuate rapidly and be influenced by ambivalence, mood and emotional state, change in perceived circumstances, level of trust, wish to protect others, shame, denial, rationalisation, coping patterns, cognitive impairment, gain, and impulsivity @xcite @xcite . there is a need to appreciate that even if increase the specificity and sensitivity of a technology capable of screening for a particular disorder, behaviour or risk, the positive predictive and negative predictive value will vary as a factor of prevalence in the targeted community. suicide is a low base rate event making prediction complex and making the capacity for undue harm and intrusive unnecessary interventions higher. even if the sensitivity and specificity of a test for suicide is significantly improved the positive predictive value may still be relatively low. this is not an argument to stop researching computational linguistics' capacity to improve suicide prediction but is a call to be cognisant of the limitations and to take a broader view on how machine learning and computational linguistics may contribute to suicide risk management. different people will have different pathways, processes, contexts, and timelines that take them to suicide. some may have a more linear escalating suicide risk chain they follow; others will display a more complex emergence pattern where multiple factors came together at a certain point in time and a chain is only apparent with retrospective coherence @xcite . while making suicide predictions has inherent complexity and limitation, enhancing the capacity for machine learning to detect risk signals and offer support to reach out and seek help, would provide a chance to positively change that pathway and context @xcite . machine learning may be able to assist in identifying what key potential contextual risk factors are for an individual or community; assist in the triage and prioritisation of attention for that individual or community; and do this at a speed and scale over multiple sources that exceeds human capabilities @xcite . ## formulation at the centre of clinical practice the clinical assessment and intervention process typically involves the key dynamic, integrated, iterative stages of history taking, mental state examination, formulation, diagnosis and care planning. formulation is at the centre, prioritising and integrating key aspects of the assessment as a foundation for the personalised intervention planning. formulation can be perceived as a form of clinical storytelling. clinical formulation includes the recurrent patterns, key themes, plot points and relationships, and cultural and contextual factors that characterise and help draw a mental model of an individual and their world. many individuals even if they have never had a formal mental health diagnosis before, may be found on assessment to have had recurrent patterns suggestive of previous episodes or prodrome or vulnerabilities. those that have an established recurrent relapsing disorder, may have patterns of risk behaviours and contexts and early warning signs, that the client has varying and fluctuating levels of insight into, but that may be well recognised by families and supports. storytelling in written and spoken language has traditionally been a way to transmit knowledge and understanding to a group and through generations. the narrative structure of storytelling may assist in human recall and motivational understanding. clinical formulation is a structured way to make sense and meaning of another's consciousness and experience and convey the story of their life, with a view to positively influencing the next stage of that story. multiple factors and models are weighed and weaved using a structured process of analysis and reporting. computational linguistics with its strengths in factor identification, weighing and integrated pattern recognition across multiple sources and contexts could assist this process. unlike describing the precise formulation or composition of a chemical compound or drug, the specific elements or processes by which a human experience arises is unknown. there is limited knowledge of the aetiology or pathogenesis of mental disorders, nor the nature of consciousness or emotions or experience. however, there are a range of biological, psychological and sociological research-based models that may guide the formulation process. the quality of the formulation is dependent on the knowledge and skill of the clinician in history taking and mental state examination and being able to identify key patterns, vulnerabilities, strengths, relationships and structures and integrate the findings with appropriate theoretical models. the quality of the formulation can be iteratively improved by the availability of additional data sources and the input of the multiple stakeholders, family, supports and caregivers that may play a role in an individual's life @xcite @xcite . ## clinical formulation structure the clinical formulation may be approached in a structured manner, with data and key findings captured under a series of interrelated headings, each capturing a different but interrelated descriptive, theoretical or explanatory perspective @xcite . machine learning and computational linguistics operating under various levels of autonomy, could assist in augmenting this clinical pattern recognition and modelling process. some variant of a series of "p" headings such as problem, predisposing, precipitating, perpetuating, protective factors and prognosis headings is common clinical practice. patterns, preferences, and priorities have been added for this work to emphasise these key attributes of the formulation process and where machine learning and computational linguistics could play a key role in capturing and weighing and providing decision support. ## ethical and regulatory issues the following section outlines a range of ethical and regulatory issues that are important considerations when developing and deploying digital mental health interventions particularly in the area of suicide prevention. health interventions should be evidence based, and subject to academic, clinical governance, regulatory and ethical review. there is a need to be ethically cognisant of the risk-benefit profile, the relative utility and costs and the numbers of people that may additionally benefit or be harmed by an intervention within a specific context. the relevance, meaning, sensitivity and specificity of screening and diagnostic tests must be described with reference to a stipulated time period, prevalence and clinical context @xcite . digital mental health interventions including computational linguistic based suicide prevention interventions need to be subject to similar standards. primum non nocere or "first, do no harm" is a fundamental principle of bioethics. failure to understand the complexity and limitations of suicide risk prediction has significant capacity to cause harm. simplistic, generalised or static risk categorisation can lead to unintended harm and there is a need for dynamic formulation based assessment that recognises the importance of context for an individual's strengths and vulnerabilities. the analytic power, reach, personalisation, timeliness and vigilance of ai based digital care affords major potential benefit. however, ai can be intrusive, discriminative, unwanted, and wrong. in suicide prevention resources could be allocated to the wrong groups, to the wrong individual, or be of the wrong type or quality and quantity. some individuals may have unnecessary protections or intrusions placed on their lives which are damaging or disabling @xcite . ai algorithms are subject to bias, misuse, undue trust, and unintended consequences and require continuous ethically based and sociocultural aware research, co-design, and governance @xcite @xcite . continually striving to improve ai based risk prediction and management at an individual to societal level and getting to zero people dying by suicide is a morally worthy goal. however, there is a need to consider how the nature and current status of attaining that goal may be societally interpreted or misinterpreted. stigma can have an impact on suicide bereavement and is an important consideration for suicide postvention. bereaved family and caregivers can experience significant stigma, shame and blame and societal judgement based on a belief that they should have seen the signals of pending suicide and predicted and prevented the death @xcite . in the reporting of improvements in suicide risk prediction, it is important that the clpsych community highlight the ongoing complexities and limitations in identifying, seeing, analysing and acting on the signals and do not unintentionally contribute to exacerbating suicide bereavement and stigma. if a digital system claims a clinical or therapeutic intervention function, then the system can be expected to be held to a high ethical and regulatory standard. this includes requiring a high level of mandated understanding of how the system integrates into broader clinical care processes, medicolegal responsibility and governance frameworks and whether it potentially requires software as a medical device type certification. there are varying developing regulatory standards and definitions for medical device type software. an ai based system providing triage and treatment advice where an individual may be at risk of suicide would likely present some of the highest ethical and clinical risks for development and deployment in a healthcare context and attract the highest regulatory categorisation and governance requirements @xcite @xcite . there is increasing interest in the use of social media and ai in suicide prevention. the international literature on social media research highlights various contentions including defining public vs private data, consent and anonymity and minimising bias and algorithmic harm @xcite . there is increasing recognition of a need for social media-based research to have ethical overview to ensure that quality research is being proposed that understands the limitations and context of the data analysis and is protective and respectful of potentially vulnerable communities @xcite @xcite @xcite . tutelary law and ethics, relates to those aspects of the legal and ethical system that have a focus on guardianship and protection @xcite . mental healthcare services have had a long and difficult history with care and protection and guardian roles. the legal system is aware that good protective intents do not always result in good or optimal outcomes and there is always a need to consider who will guard the guardians. clinical decisions and opinions about risk that impact on an individual's civil liberties, are often subject to review by tutelary mental health courts and tribunals; decisions influenced by ai based categorisation or predictions should similarly be expected to be reviewed by the tutelary system @xcite . @xcite have argued ai ethics can be reduced to five core principles. four of these are the traditional bioethical principles of autonomy, justice, beneficence and non-maleficence to which they have added explicability. explicability aims to capture the concepts of intelligibility and accountability. building suicide prevention interventions and research on faulty, limited or poorly understood or described models affords significant ethical and clinical risk. clinicians and researchers need to take a lead in ethically shaping and governing the emergent capacity for greater levels of social media and ai based suicide prevention research and development @xcite @xcite . before the deployment of ai in a mental health setting, stakeholders should have an adequate understanding of how it was co-designed and works and who is accountable and liable for how it works @xcite . the formulation process could improve explicability in that there should be a clearer, intelligible and accountable process as to why intervention decisions were made. this should include having an understanding of the mental health theoretical models on which or for what, the machine learning algorithms were built (de @xcite . ## discussion and conclusion in clinical practice there are significant standardisation, ethical, safety and effectiveness issues when classifying an individual as in or out of some binary diagnostic or risk category. this is particularly so when the constructs or models that underlie each criterion are limited in their scientific basis and are not operationally defined and there is significant variation in training, interpretation and application and perceived clinical utility. similarly, when computational linguistics developments aim to assist in symptom, diagnostic and suicide risk prediction categorisation there may be significant theoretical, ethical, utility and clinical safety concerns and limitations. there is a need to move beyond risk categorisation to risk formulation as part of the broader clinical formulation and intervention context. any risk prediction categorisation produced needs to be treated like the output from any screening or diagnostic test; that is as another datapoint for the formulation, that is to be iteratively weighed, integrated and interpreted within the broader dynamic clinical context and not considered definitive or static. human experience is often time and context dependent, dynamic and multidimensional and occurs along a spectrum rather than within discrete categories. formulation is the key focus of natural clinical intelligence and ought to be a key focus for artificial intelligence. computational linguistics could help in the de-velopment and assessment of a broader contextual understanding of an individual's history and mental state. a diagnostic and risk formulation process affords the opportunity to present a richer personalised explanatory model that links all the factors and highlights the complexity, uncertainty and importance of context and dynamic change. the clinical formulation process of iterative factor identification and weighing, pattern recognition and modelling, is in keeping with the strengths of machine learning and the computational linguistics process. there are opportunities for significant synergy. computational linguistics can operate at a speed and scale of factor identification and analysis across multiple sources beyond human capability. the machine learning process may be refined on previous clinical assessments, with emphasis given to mental state examinations and formulations. clinician in the loop training and curation processes may assist with explicable and reflexive algorithmic improvement and production of meaningful safe ethical outputs. though such formal clinical data may be difficult to access for current researchers, this can be expected to improve as machine learning is integrated and normalised as part of care delivery. machine learning and computational linguistics could improve the explicable quality of the acquisition, analysis and description of formulation data. machine learning could also improve the quality of the theoretical models applied by improving the quality of research that underlies those models. there may be different and changing reasons and typologies for suicide and ai enabled research may be able to better timely categorise, trend and define these at an individual and community level @xcite . in formulation every current and emergent finding needs to be iteratively analysed in context, and with knowledge of the strengths and limitations of the related clinical theoretical models. particularly in suicide risk management there is a need to be highly cognisant of the difficulty and contention of predicting complex low base rate events and the harm that can result from both false negatives and false positives. digital transformation and co-design in ai empowered suicide prevention requires the working together of clinicians, communities, consumers, and digital media companies. the leverage, reach and analysis of ai empowered digital media make taking a co-design and societal perspective more meaningful and achievable and anything more limited, less ethically justifiable. looking to the future computational linguistics could assist in the creation of a self-constructing and updating digital formulation drawing on multiple sources from social media to email to clinical notes and assessments to conducting autonomous interviews in oral and written format. these services could be delivered in the form of customisable digital guardians, coaches or clinicians that address, with varying levels of expertise, medicolegal responsibility and autonomy, the assessment, formulation and intervention process. in terms of an example of potential next research steps the authors are currently integrating qualitative thematic analysis with machine learning based topic modelling to study sleep related concerns in a large social media based suicidality dataset. sleep disturbances from insomnia to nightmares to sleep disordered breathing are associated with an increased risk of suicidal behaviour and night-time is a high-risk period for suicide @xcite @xcite @xcite @xcite @xcite . the research is exploring whether this integrated thematic analysis and topic modelling approach can contribute to the development of an explicable conceptual linguistic sleep signal model for ai empowered clinical formulation, prioritisation, treatment category recommendation, and psychoeducation in the area of suicide prevention. identifying key topics and themes and a related lexicon are central to these clinical processes. suicide is complex and multifactorial. by focusing on one potential signal (sleep), one machine learning technique (topic modelling) and one dataset the aim is a greater conceptual understanding of the opportunities and challenges that could be presented by an multi-signal, multi-source, explicable ai and formulation based suicide prevention system. the current focus is on social media data, but a range of biopsychosocial data sources might be integrated into a future system. formulation and broader data contained in clinical assessments and discharge summaries could be a future key target for both analysis and enrichment @xcite . developments in digital formulation from a clinician augmentation or decision support role to a more autonomous social media focused digital guardian, coach or clinician role will have major clinical and societal impact. suicide is a complex time and context dependent phenomenon. there is increasing recognition of the need to broaden the clinical service focus of suicide prevention to a more societal level focus that has more timely vigilance and leverages a greater range of resources. clinic-based services accessing social media data, and social media based services accessing formal clinical data, and the integration of such services raises significant medicolegal, security and ethical issues @xcite @xcite . however, machine learning assisting in the expansion from a clinical service to societal focus allows for more protective layers and opportunities for integrated formulation and intervention at more time points. social media machine learning based interventions have the advantage that even if they have only a relatively small effect size on reducing suicidal behaviour they can be deployed at such scale and minimal marginal cost that they may have a significant impact at a population societal level @xcite . socialising emergent concepts, among research and practice leaders, is an important stage in the innovation diffusion and health practice change process @xcite @xcite . this can lead to critical analysis of relative advantage, adoption challenges and health impact , and feed through to strategic research, implementation and governance plans @xcite . this paper has aimed to socialise the concept of digital psychological formulation with the goal of making a positive health impact on suicide prevention by promoting adoption and development of the concept by the clpsych community. there is a significant concept reality gap between current developments and getting to a stage of digital formulation being utilised by human and digital clinicians as part of standard mental health and suicide prevention practice @xcite . however, it is a concept reality gap that is potentially fast narrowing and that the clpsych community has both the developing expertise and ethical duty to take a leadership role in crossing, in a safe and clinically effective manner.