Safeguarding against spurious AI -based predictions: The case of automated verbal memory assessment (2021)

Abstract: A growing amount of psychiatric research incorporates machine learning and natural language processing methods, however findings have yet to be translated into actual clinical decision support systems. Many of these studies are based on relatively small datasets in homogeneous populations, which has the associated risk that the models may not perform adequately on new data in real clinical practice. The nature of serious mental illness is that it is hard to define, hard to capture, and requires frequent monitoring, which leads to imperfect data where attribute and class noise are common. With the goal of an effective AI-mediated clinical decision support system, there must be computational safeguards placed on the models used in order to avoid spurious predictions and thus allow humans to review data in the settings where models are unstable or bound not to generalize. This paper describes two approaches to implementing safeguards: (1) the determination of cases in which models are unstable by means of attribute and class based outlier detection and (2) finding the extent to which models show inductive bias. These safeguards are illustrated in the automated scoring of a story recall task via natural language processing methods. With the integration of human-in-the-loop machine learning in the clinical implementation process, incorporating safeguards such as these into the models will offer patients increased protection from spurious predictions.

## introduction artificial intelligence (ai)-based systems that incorporate language and behavioral data hold promise of increasing sensitivity, equity, and access in the assessment and treatment of mental illness through the use of remote and continuous monitoring via clinical decision support systems. this is due to the fact that the pattern and content of language, as well as additional measures of behavior, such as timing and neuropsychological task scores, provide rich information that can be traced back to an individuals' overall mental state. in order to demonstrate clinical translational value there are numerous risks and factors that are necessary to consider. first, it is important to collect data from large samples of the population across differing ages, cultures, genders, clinical conditions, and stages of disorder. second, it is critical to create models that are explainable, transparent, and generalizable @xcite in order to nurture trust from both patients and clinicians. and finally -the area that this paper will address -it is necessary to add safeguards to models such that they are capable of flagging cases that show attribute noise (i.e., abnormalities in feature values) or class noise (i.e., erroneous or missing class labels), and of determining the extent to which models will generalize to unseen data. these safeguards will enable a human-in-the-loop system where humans are required to review data abnormalities. ai is used in a wide range of applications within mental health, notably within clinical research settings where data are used to aid in understanding the nature of diagnoses and to improve diagnostic accuracy (for reviews see @xcite @xcite , as well as in making complex and potentially lifesaving de-cisions (e.g., in suicidology -for review see @xcite . acoustic measurements of speech have been analyzed in automated applications for detecting mild cognitive impairment and dementia @xcite , as well as serious mental illness @xcite and depression @xcite . in the domain of techniques that specifically leverage natural language processing (nlp), there are a growing number of reports of using these methods on social media data, notably to data mine publicly shared written reports of mood on platforms such as twitter and reddit @xcite @xcite . there is also a growing interest in using such methods on electronic medical records to assist in the extraction of diagnostic information or to enhance understanding of medical conditions @xcite @xcite . a broad range of nlp metrics such as incoherence and tangentiality have been used to automatically assess the clinical state of patients with schizophrenia @xcite and predict the risk of psychosis onset @xcite @xcite . deep language models and nlp featurebased models have also been shown to differentiate the language of healthy controls from those diagnosed with mild cognitive impairment or dementia @xcite . there is clear evidence that the clinical data used in ai-based research applications hold predictive power in detection and diagnosis, prognosis, support and treatment, and as a second opinion measurement for illness severity, but it is unclear about the degree to which these models will be stable on new data. many psychiatric studies that harness ai tend to do so on relatively small datasets (i.e., 10-100 participants) in fairly homogeneous populations (e.g., the weird (western, educated, industrialized, rich, and democratic) phenomenon - @xcite ; and the predominance of male participants in psychiatric research studies @xcite . these shortcomings may lead to insufficient accuracy on unseen data retrieved from different experimental settings (e.g., in a lab vs. remote; prompted free speech vs. natural; as a component of a larger testing battery vs. on its own), populations (e.g., southern vs. northern; different english speaking countries; monolingual vs. multilingual participants), and clinical states (e.g., hallucinating vs. not hallucinating). one must keep in mind that in small datasets, spurious features may not be generalizable to a larger population, especially if they are not of any apparent clinical relevance @xcite . while these research experiments are noteworthy, they must be re-evaluated on larger and more diverse sets of participants to test for robustness and generalizability. incorrect or ill-advised decisions and predictions in psychiatry can be dangerous and life altering for patients, and the difficulty in decision making is further confounded by the very short time frame in which changes in mental state occur and the associated clinical decisions must be made. thus, we must build systems that have the ability to instantaneously flag data abnormalities -both in the research phase and when translated into real clinical use -and pass these cases on for human review. furthermore, rather than selecting a preferred machine learning model based on metrics such as accuracy, sensitivity, or correlation as is common in ai and nlp applications, we must seek to understand the underlying mechanisms and the context in which they will be used @xcite . researchers in machine learning have proposed assessing models with stability metrics which define ways to quantify and compare the stability of results rather than simply focusing on the aforementioned metrics @xcite . specifically, @xcite differentiated data-based noise and outliers into class noise and attribute noise, and advocated for analyzing their effects on machine learning models separately. uncertainty estimation, as well as inand out-of-distribution error detection has been critically important in the use of ai in a wide range of applications such as self driving cars @xcite , general medicine @xcite , education @xcite , and in many other domains. in this paper we illustrate an example of nlp and machine learning methods applied to the automated scoring of a story recall task, a core component of psychiatric neuropsychological assessments. we focus on two approaches to safeguarding such a model: 1) the detection of attribute and class noise that can affect the predictions of a model and 2) the evaluation of the extent to which the model may or may not generalize to unseen data. we first applied methods to determine where noise exists with an outlier detection algorithm and data visualization. for the issue of model generalizability, we studied the effect of dataset size on the results, and we illustrate how such results change as we randomly remove portions of our training data. additionally, we show the results of this particular story recall model applied to a new collection of data. we advocate that these computational safeguards, which have major implications in regard to their use in human-in-the-loop clinical support systems, must be placed on each machine learning model that is developed to automate or assist in clinical assessments. ## effects of attribute and class noise we begin our analysis of computational safeguards by discussing the determination of attribute and class noise in the context of model stability. model stability analysis allows us to establish how unusual variations in input data will affect the output of the model. put simply, we wish to find where in the feature space models may be the most unstable. we illustrate an approach that will allow researchers to detect attribute and class noise in data that could be due to construct-irrelevance or errors in assumptions. specifically, attribute noise is where values of individual attributes do not make sense; whether they are erroneous or missing. class noise is where a label does not make sense given the distribution of the features for other data with the same label; whether it is mislabeled or contradictory. in order to make the notions of attribute noise and class noise concrete, see our first outlier analysis was based on researchstage settings where we have access to both attribute values and class labels. while this exact approach may not always be feasible in the eventual clinical application stage (since there are not always ground truth class labels available), the approach itself can nonetheless be harnessed in the same manner but with the omission of ratings, classes, or labels. here, we discovered outliers using the isolation forest algorithm @xcite . most outlier detection algorithms first find the normal region of data and subsequently define anything outside of this defined region to be an outlier. the isolation forest algorithm, on the other hand, discovers minority data points that have attribute values that differ from those of the usual instances. specifically, the algorithm isolates examples by selecting an attribute at random and then selecting a random split value between the maximum and minimum values of the selected feature. anomalous examples will have shorter paths from the root to the leaves in their isolation trees than the normal examples since they need fewer partitions to be isolated. this algorithm is well-suited for high dimensional datasets and has proven to be an effective way of detecting outliers and anomalies @xcite . furthermore, it works especially well for behavioral data as "normal" regions tend to be more variable than in other domains. the current outlier analysis was specifically based on the number of types (i.e., unique words), the number of common types between the original story and the recall, the bertscore between the original story and the recall, and the human rating given to the recall. figure 1 shows the results of applying the isolation forest algorithm to the story recall data. it is shown that 18 outliers were detected. such instances would be flagged for human review, where researchers can determine if attribute or class noise is present and either fix the erroneous values or exclude them from the modeling in the case that the examples are entirely invalid. when the approach is used in clinical settings to flag attribute noise, clinicians can review the raw data and make determinations for themselves rather than relying on a machine prediction. out of the 18 examples flagged by the isolation forest algorithm, 9 were determined to be invalid responses (i.e., participants stating that they simply do not remember or responses that are insufficient for data analysis) and 9 were valid responses with either sparse amounts of language or large amounts of language but poor performance. the average absolute error on the outliers was 1.34 (@xmath0.80); the valid response outliers generated a higher absolute error (@xmath1.63, @xmath2.91) than the invalid response outliers (@xmath3.05, @xmath4.63). the performance of the model on outlier data is far lower than the models overall performance. as the contamination threshold of the isolation forest algorithm is increased (i.e., the criteria for an outlier is relaxed), additional responses are chosen that mirror the behavior of these 18. this is a parameter that would need to be tuned such that all true outliers are detected yet it does not extend into the normal data range. furthermore, this parameter will need to be learned by investigating the true distribution of these phenomena and will depend on the application. interestingly, the model performance did not change with the removal of these outliers. as approximately 2% of the data was flagged in this experiment, the model behaved indifferently to their exclusion. the exclusion of extremes (which help the performance of the model) combined with noise (which harm the performance of the model) potentially balanced out the effects of both. this isolation forest analysis can be performed on the same data without ratings in the eventual clinical stage to find attribute noise and extremes. we also present an analysis of features alone that can be done in any stage of the modeling process. a basic noise detection approach that can be used at any stage of the modeling process is to simply find the examples with low attribute agreement (assuming that the attributes are collinear). figure 2 depicts the distribution of two of the most predictive features of the story recall rating prediction model (the number of common word types and the bertscore between the original story and the recall). there is a steady agreement between the two features, with some outliers (marked with crosses) outside of the diagonal where the features do not agree. the color of the circles represent how far off the model rating was from the human rating. two examples with exceptionally high error (~2.5-3.0) are identified in red. the bottom-most red example is a response with a mixture of correct and incorrect (random) details, as well as incoherent language. the top-most red example is a response with a high bertscore even though only a recitation of the title of the story was spoken. this high disagreement between features in turn uncovered a faulty feature score potentially due to flawed weighting parameters in the bertscore model. we have shown that examples located off of the diagonal in plots such as these should be passed on for human evaluation as disagreement in two objective collinear attributes of story recall may raise concern. finding these outliers is critical because if a model has not been exposed to certain combinations of features or labels in its training set, then we cannot assume that it will produce accurate predictions in such settings. outliers are important to detect both in the research stage in order to update or exclude certain examples from affecting the model in a negative manner and in the clinical setting so that spurious decisions are not made on abnormal data. ## effects of model generalizability as previously stated, one of the most critical safeguards to spurious ai-based predictions is using large, diverse, and representative data @xcite , but this is not always possible. when using human behavioral data in machine learning algorithms, researchers inadvertently make the assumption that there is one canonical representation of specific groups of humans (i.e., those with serious mental illness), but this is simply not true. those with psychiatric disorders exhibit extremely diverse symptoms and behaviors. human behavior displays patterns indicative of a chaotic system @xcite , which holds true for behavior within one person as well as behavior within a group. to approach the topic of generalizable data, we first explored whether choosing different subsets within a training dataset would affect the output of the resulting model and whether there are spurious results when using smaller subsets. the story recall regression model was trained on @xmath5, a large size relative to clinical experiments in the mental health domain. we used stratified sampling to create smaller subsets of the data that retain the proportions of each rating and tested how the model behaved on these smaller subsets. since experiments based on subsets of data retrieved from the same experimental population and setting do not necessarily show the true extent of model generalizability, we also performed transfer tests of the story recall model. specifically, a second dataset was collected from inpatients at a substance abuse program in louisiana (@xmath6, most of whom suffered from co-occurring mood, psychotic, anxiety and personality spectrum disorders, as well as an additional collection from presumed healthy undergraduates at louisiana state university (@xmath7. together, the inpatients and the presumed healthy undergraduates completed @xmath8. a ridge regression model with the same nlp features as previously reported was trained on the initial dataset and tested on the new dataset, as well as vice versa. the first experiment resulted in a pearson r correlation of 0.86 and the reverse an r of 0.84. here, we conclude that the story recall regression model will generalize to differing clinical populations as well as illness severities. the same may not hold true for differing cultural populations as language differences may prove to be a confounding variable in transferring such a model. we thus advocate testing models on each new population prior to implementation. neuropsychological task scoring is a much more objective application area than other modeling applications in this field in which less is known and gold standard labels are often disagreed upon (e.g., disease detection, mental state tracking, and so on). thus, generalizability is much more critical to test in these other applications and will potentially not yield such robust conclusions. nonetheless, the understanding of when a model will yield accurate output and when it will not is an extremely important endeavor. finding representative data is of the utmost importance in machine learning. in some cases, such as the story recall regression model, it is best to get as much data from as many people as possible. in other cases, especially when dealing with extreme diversity between individuals or subsets of individuals, it may be best to only use data that behaves in a similar fashion to the example currently being tested. ## discussion mental health is extremely dynamic as it can change on the scale of seconds, minutes, hours, or days, and language offers an objective and potentially unobtrusive way to assay such changes. mental state in some conditions can change quickly with fatal consequences (e.g., suicide attempts) and more frequent monitoring of language and behavioral data, combined with machine learning methods, has the potential to offer clinicians unprecedented support in tracking patient state. language can be harnessed for many applications as it offers a quantitative conceptualization of a person's underlying thought processes and mental health. tracking such phenomena is extremely important yet increasingly complex, and as such there is a need for greater reliance on model outputs in this field. in experiments involving nlp methods, it is common to deal with high dimensionality from features like word embeddings, parser outputs, and so on, which makes interpretation and understanding of models difficult. features often go beyond normal distributions and as such there tends to be high variability in data distributions. thus, it is especially important to create methods and tools that allow us to better understand the feature space and determine whether attributes or classes may violate assumptions. an eventual goal of this line of work is to have a human-in-the-loop system where models analyze streams of high dimensional patient data and produce predictions of mental state and well-being. in the research stage of this implementation, real data must be analyzed to determine what normal distributions of attributes and classes appear to be. aberrant instances of patient data can be flagged and reviewed by researchers to either update or exclude from models. researchers must also test their models' generalizability by collecting additional samples or performing validation techniques to verify performance on unseen data. this process will allow for models to be based on the most accurate and representative data. in the eventual clinical decision support system implementation, models must be realized such that attribute outliers are not predicted on, but rather the raw data is passed to a clinician to make a judgment. if the outlier is due to faulty feature values, clinicians can update these values or they can create their own labels and update the system such that future similar cases would not necessarily need to be verified by a human. in such a situation, there is a "best of both worlds" where models can execute the tasks that they are best at (high dimensional data analysis) and humans can execute the tasks that they are best at (handling anomalies and interpreting patient data). for nlp and machine learning methods to be adopted in current research experiments as well as in eventual clinical practice, they require critical peer evaluation. what is needed is transparency in terms of data collection, validation, reproducibility, and clinical agreement in the association of language features to underlying illness. this paper showcases how essential it is that clinicians are involved in all stages of development. as such, it is a large step towards bringing more ethics and transparency into ai-based studies in mental health. ethics review boards must demand this type of transparency and fairness in the creation of models so that systems that harness machine learning can be implemented in real clinical practice with low risk. some discussion of this path forward has been brought to light by @xcite who reported on irbs as a means of ethics oversight in health research that harnesses ai. ## conclusion this paper illustrates the importance of understanding the assumptions and distributions that underlie training data and the algorithms used, as well as the need to flag data that have characteristics that violate these assumptions. not only is this knowledge important, but so too is having the tools to do this. we found model instabilities in a story recall regression model with the use of outlier detection algorithms and error analyses with respect to varying input. we advocate that approaches such as these be incorporated into machine learning and nlp-based clinical research and implementation. with the complexities inherent to models based on many features, high numbers of parameters, highly variable human behavioral data, and extremely high (and potentially fatal) stakes for mistakes, it is critical to establish methods beyond model designer intuition in assuring robustness and that predictions cannot be made on out of range data or data that lies in areas of instability. it should now be obvious that high predictive power on a relatively small dataset does not entail clinical relevance or generalizability, and that it is essential to use larger data sets, have more data collection outside of controlled settings, incorporate modeling safeguards, and use human-in-the-loop methodologies at all steps of the process.