Distinguishing between Dementia with Lewy bodies ( DLB ) and A lzheimer’s Disease ( AD ) using Mental Health Records: a Classification Approach (2020)

Abstract: While Dementia with Lewy Bodies (DLB) is the second most common type of neurodegenerative dementia following Alzheimer’s Disease (AD), it is difficult to distinguish from AD. We propose a method for DLB detection by using mental health record (MHR) documents from a (3-month) period before a patient has been diagnosed with DLB or AD. Our objective is to develop a model that could be clinically useful to differentiate between DLB and AD across datasets from different healthcare institutions. We cast this as a classification task using Convolutional Neural Network (CNN), an efficient neural model for text classification. We experiment with different representation models, and explore the features that contribute to model performances. In addition, we apply temperature scaling, a simple but efficient model calibration method, to produce more reliable predictions. We believe the proposed method has important potential for clinical applications using routine healthcare records, and for generalising to other relevant clinical record datasets. To the best of our knowledge, this is the first attempt to distinguish DLB from AD using mental health records, and to improve the reliability of DLB predictions.

## introduction alzheimer's disease (ad) is the most prevalent type of dementia, characterised by progressive cognitive impairment such as memory loss. dementia with lewy bodies (dlb), also known as lewy body dementia, is the second most common type of neurodegenerative dementia following alzheimer's disease (ad), with the defining features of fluctuating cognition, recurrent visual hallucinations, rapid eye movement (rem) sleep behaviour disorder, and parkinsonian motor symptoms in addition to dementia @xcite . particularly in the early stages, prior to diagnosis, dlb and ad are difficult to distinguish, hence the detection rates of dlb are sub-optimal, with a large proportion of cases missed or misdiagnosed as ad @xcite . detection of dlb is, however, crucial as compared to ad and other forms of dementia (e.g. parkinson's disease dementia (pdd) foot_0 ). dlb has a worse prognosis across key outcomes such as mortality, hospitalisation, move into residential care, quality of life, and healthcare costs @xcite . moreover, not only is early diagnosis paramount, different types of treatments can have different impacts on these patient groups, e.g. antipsychotics, which adds to the importance of accurate and timely diagnoses. due to the challenges in recognising dlb clinically, it has been difficult to recruit large research cohorts of representative patients with dlb, and the increasing use of routinely collected healthcare data has been suggested as a potential solution to this shortage. applying classical methods of symptom ascertainment using natural language processing (nlp) in routinely collected data is however difficult in patients with dlb, as clinicians tend to record the defining features only if they have also made the correct dlb diagnosis @xcite . therefore, we applied novel neural models of nlp to test whether these can be clinically useful to distinguish dlb and ad, and to provide assistance to mitigate expensive outcomes from misdiagnoses of dlb. this task is challenging because dlb and ad share certain clinical and biological similarities that make them particularly difficult to differentiate. motivated by the emergence of neural models and nlp methods applied to the biomedical domain, we cast this as a binary text classification task, where we use convolutional neural networks (cnns) @xcite @xcite to address it. additionally, the generalisation of well-trained models is notably more difficult, since different formats and grammatical patterns emerge in mhrs across different healthcare institutions. in order to test the efficiency of our proposed methodology, we use three datasets from two different mhr (clinical documentation) systems and healthcare institutions, with the aim of comparing the model's performances on similar datasets containing relevant data, but with different contextual structures. to assist the analysis of our experimental results, and to bridge the gap between model accuracy and confidence, we also study an approach where the model confidence estimates are calibrated. confidence calibration is important for classification models. classification networks must not only be accurate, but should also indicate when they are likely to be incorrect; a well-calibrated network matches its confidence to its accuracy so that it is confident when it is accurate, and uncertain when it is not. we use the calibration method named temperature scaling, where expected calibration error (ece), the expectation of the differences between confidence and accuracy, is used as the primary empirical metric to measure calibration @xcite . in this paper, we present our preliminary work towards automatically distinguishing individuals diagnosed with dlb or ad using neural network models and mhr texts. this methodology can provide an efficient technique for detecting and intervening dlb. our contributions are threefold: 1) we introduce a cnn approach for the classification on dlb and ad using mhrs; 2) we investigate the performance of the proposed model on two mhr datasets from two different healthcare institutions with different formats and patterns; 3) we also apply a neural model calibration method to help in understanding when the model predictions tend to be brittle, so that the model can output confidence scores with higher reliability. ## related work with the success of neural models for many nlp tasks, deep learning methods, as well as word embeddings, have started to be applied to the biomedical and/or clinical domains @xcite @xcite including mental health, such as automatic detection and classification of cognitive impairment. for example, three neural models (cnns-, lstm-rnns-, and cnn-lstm-based) were applied to distinguish ad and control patients from dementiabank @xcite . cnn-lstm model achieves state-of-the-art performance on the ad classification task. since neural models are usually black-boxes and it is hard to interpret the reasoning for final classification decisions, various visualisation techniques have been proposed for neural networks @xcite @xcite . @xcite illustrated two visualisation methods for interpretation, based on activation clustering and first-derivative saliency methods, to assist the analysis and consolidation of distinctive grammatical patterns of contextual information from ad patients. early detection plays a crucial part in the study of dementia. @xcite proposes a hierarchical model that encompasses both the hierarchical and sequential structures of picture description with attention mechanism, and detecting signs of cognitive decline at both the word and sentence levels, by using the dementiabank and an in-house database of cookie theft picture descriptions @xcite . @xcite shows both the proposed hierarchical structure and the attention mechanism contribute to the improvement in ad detection. most nlp studies addressing dementia use language transcripts from clinical cohorts, such as the dementiabank @xcite . to our knowledge, very few studies have used mhr documents and nlp for modelling detection of dementia types, and we are not aware of any studies using nlp and mhrs for detection of dlb. mccoy jr. et al. (2020) presents a study using electronic health record (ehr) data for stratifying risk for dementia onset, using a bespoke nlp approach for scoring symptoms in the clinical texts. this nlp approach, however, relies on pre-defined terms, and addresses a slightly different clinical problem. when applying neural networks to real-world decision-making systems, classification networks must not only be accurate, but also should indicate when they are likely to be incorrect. a network should provide a calibrated confidence measure in addition to its prediction. calibrated confidence estimates are also important for model interpretability. @xcite identify methods, which can alleviate miscalibrated problems in neural networks, and offer insight and intuition into network training and architectural trends that may cause miscalibration. good confidence estimates can provide valuable extra information to establish trustworthiness in early detection of cognitive impairment. ## methodology our proposed approach uses a cnn model to distinguish dlb and ad patients. we compare the performance of using an embedding layer (emb-layer) and pre-trained embeddings (bioword2vec) on our classification task, and finally apply a post-processing method (temperature scaling) for model calibration. ## materials and experimental setup by applying two types of word embeddings (emb-layer and bioword2vec) for word representations, convolutional neural network (cnn) for model training, and temperature scaling for model calibration, we investigated and evaluated the efficiency of our proposed methodology on three datasets from two healthcare institutions. ## results comparing the performances of random initialised word embeddings (emb-layer) and pre-trained bioword2vec, the result using emb-layer achieves higher f1-score (0.87) than bioword2vec (0.63) for cris. results on crate using emb-layer and bioword2vec are, on the other hand, quite close considering f1scores and their stabilities for 5-fold cv. however, for cris † , using pre-trained embeddings bioword2vec (0.78) performs better than emb-layer (0.73), with more comparable data sizes of dlb and ad. (0.77) is slightly higher, the crate has a comparatively lower deviation. this result might be inherited from the fact that there is a significant increase in the overlap between cris † and crate datasets (see we also report the top-20 most important features contributing to the prediction in the lr model using bow and tf-idf counts representations (see inspired by the important features from lr, our baseline method, we removed the top-ranked important words from the pilot training data. we observed that after removing the core dementiarelated words we still obtain similar f1-scores for crate using both types of embeddings (see using model a as the base model for model calibration, where raw text serves input to our cnn model with the bioword2vec word representation, we obtain well-calibrated model for all cris, cris † , and crate (see it is worth noting that models trained on three datasets experience some degrees of miscalibration. ## discussion to our knowledge, this is the first study on automatically distinguishing dementia with lewy bodies (dlb) from alzheimer's disease (ad) using mhrs. we investigated the performance of cnn models using different embedding representations on mhrs from two different healthcare institutions, and incorporating the method of model calibration into dlb classification to obtain reliable predictions. to be able to apply nlp models to real-world biomedical tasks, we need first to embrace the challenges of the datasets. in our case, we face a range of such challenges: small data size, hence reduced reliability of predictions; class imbalance; noisy data; and contextual differences between datasets. these might be the reasoning behind higher deviation and instability of f1-scores observed in some predictions (see we attempted to mitigate these challenges by using a set of fairly standard techniques. we use 5-fold cv to ensure that every example appears during both training and testing. using 5-fold cv, important information is more likely to be learnt, and consequently obtaining better approximations and enhancing robustness, whereas with larger datasets there is more chance to have a proper distribution of information for both training and testing. since most mhrs are written with different formats and grammatical patterns, we considered using pre-trained biomedical word embeddings (bioword2vec) to get a unified word representation across different datasets. those embeddings helped our models to rely less on explicit indica- our proposed model and calibration method could prove useful clinically. currently in clinical care there is a high level of under-diagnosis as well as lack of confidence in making a dlb diagnosis. moreover, appropriate treatment is crucial, e.g. it is important to avoid antipsychotic prescribing for this patient group. although the f1-scores and calibration results are not always perfect, they indicate that using routine healthcare data could be valuable for predictive model development even in cases where it is hard to obtain large datasets. ## conclusion in this paper, we propose to use a cnn approach for the task of detecting dlb patients by distinguishing them from ad patients. our wellcalibrated models are relatively robust after using temperature scaling, where calibrated probabilities are more informative of good probability estimates and true predictions. the proposed model is investigated on two mhr datasets from two different healthcare institutions, and achieves competitive results using two types of embeddings (emb-layer and bioword2vec). the pre-trained biomedical word embeddings (bioword2vec) are effi-cient for all three datasets whilst cris relies much more on in-domain word distributions. in particular, bioword2vec can achieve lower deviations on model performance in ablation study. future work will be focused on the effectiveness of contextualised embeddings for a more general methodology where the detection of dlb can be realised across healthcare institutions. we would also like to investigate more effective pre-processing techniques to purify and clean the raw texts before feeding into advanced models, and to mitigate the noise commonly existed in health records. the code is available at https://github.com/ zixuwang1996/dlb_ad_classification .