Training Models on Oversampled Data and a Novel Multi-class Annotation Scheme for Dementia Detection (2023)

Abstract: This work introduces a novel three-class annotation scheme for text-based dementia classification in patients, based on their recorded visit interactions. Multiple models were developed utilising BERT, RoBERTa and DistilBERT. Two approaches were employed to improve the representation of dementia samples: oversampling the underrepresented data points in the original Pitt dataset and combining the Pitt with the Holland and Kempler datasets. The DistilBERT models trained on either an oversampled Pitt dataset or the combined dataset performed best in classifying the dementia class. Specifically, the model trained on the oversampled Pitt dataset and the one trained on the combined dataset obtained state-of-the-art performance with 98.8% overall accuracy and 98.6% macro-averaged F1-score, respectively. The models’ outputs were manually inspected through saliency highlighting, using Local Interpretable Model-agnostic Explanations (LIME), to provide a better understanding of its predictions.

## introduction dementia is a condition characterised by impaired memory, thinking or decision-making ability that interferes with daily activities @xcite . this global issue affects approximately 50 million individuals, with projections suggesting that the number will increase to 139 million by 2050 @xcite . while no known cure for dementia currently exists, early diagnosis is essential, as it enables patients to access interventions that can help manage symptoms, prevent further degeneration and improve their quality of life. recent research suggests that language changes and a decline in episodic memory may serve as an essential signal for early diagnosis of dementia, with language impairments reported in both preclinical dementia and severe cases @xcite . methods for natural language processing (nlp) can help in detecting dementia through the analysis of the language used by a patient of interest. indeed, previous research cast dementia detection as a binary text classification task, categorising a patient as exhibiting dementia or not, based on their language use @xcite @xcite @xcite . however, thus far, no studies have investigated the classification of patient conversation transcripts into more than two classes. our study aims to address this gap and seeks to analyse patients according to three classes: healthy control (hc), early stage or mild cognitive impairment (mci) and dementia. the goal is to provide medical professionals with a tool (that can be used in conjunction with standardised tests) for identifying patients exhibiting early-stage dementia symptoms. such a tool can be useful in organisations where there is a lack of expertise among personnel responsible for screening patients, for the purposes of identifying those who could benefit from interventions that might potentially slow the progression of the disease. our approach involves analysing speech transcripts from doctor-patient conversations, with participants categorised into the three aforementioned classes. this task is a multi-class classification problem, which we address by developing models that are capable of classifying text (i.e., the transcripts) according to three classes. in particular, we developed models based on the transformer architecture @xcite , considering that transformers have demonstrated state-of-theart performance in many clinical text classification tasks @xcite . additionally, we utilised explainability techniques to identify words that are indicative of dementia and may be used as features in the diagnostic process. recent work on dementia detection has been underpinned by text classification models based on transformer architectures. ## methodology this study employed two distinct approaches to developing dementia classification models. the first approach aimed to ensure comparability with previous research by solely utilising the pitt dataset. however, the original pitt dataset was highly imbalanced (with 259 hc, 127 mci and 24 dementia samples in the whole dataset), containing a limited number of confirmed dementia cases, necessitating oversampling to address this limitation. specifically, we oversampled the mci and dementia classes to allow for a more balanced representation of these classes in the training set. utilising stratified 10-fold cross-validation (cv) in our experiments, the resulting training dataset for each fold included original hc samples, mci samples duplicated thrice, and dementia samples duplicated 16 times. on the other hand, the test set (for each fold) was left unaltered. the second approach involved combining the pitt, holland, and kempler datasets to increase the representation of naturally occurring dementia in the dataset, thus eliminating the need for oversampling. this approach enabled us to assess the performance of the models with unique dementia data samples and a wider range of discussion topics. ## evaluation and results the objective of the experiments conducted was to test two fundamental hypotheses. firstly, it was hypothesised that utilising the novel three-class labelling system would improve classification performance by enabling a more refined classification that can distinguish between more nuanced differences in the data. secondly, it was hypothesised that models developed utilising the combination of datasets would exhibit superior performance to those developed using solely the pitt dataset. the rationale behind this hypothesis was that the combined dataset would provide a more diverse and representative range of data, ultimately improving the generalisability of the models. as described above, to test these hypotheses, three models were created using bert, roberta, and distibert for each approach. the performance of the models was then evaluated using stratified 10-fold cv, with the performance metrics being accuracy, micro-and macro-averaged f1 scores, and, importantly, precision for the dementia class. the lattermost metric is crucial in a medical diagnosis scenario: false positives for the dementia class should be minimised as they could lead to unnecessary interventions or distress. ## discussion the three-class annotation scheme improves classification performance. as can be seen in tables 1 and 2 , using a three-class labelling system improved the performance of all models. the improved performance is likely due to the finergrained system allowing for a more nuanced classification, distinguishing between cognitive impairment levels. the results demonstrate that almost all threeclass models achieved an average precision of 100% for the dementia class. the best models were able to correctly identify positive cases without generating any false positives, making them valuable in medical diagnosis. in order to provide a more comprehensive evaluation of the class-level performance of our top-performing model in terms of f1-score, the distilbert model trained on the combined dataset, a detailed breakdown of its performance table is presented in oversampling is a viable method to improving a dementia detection model's accuracy and macro-averaged f1-score. as can be seen in the addition of a small number of dementia samples from outside the pitt dataset significantly improves macro-averaged f1-score and accuracy. the best-performing model, in terms of macro-averaged f1-score, is the distilbert model generated using the combined dataset; this shows that a model using the three-class labelling system can exhibit optimal performance simply with the addition of a small number of dementia samples. although the work by @xcite did not provide any detailed performance breakdown for each class that would facilitate straightforward comparisons, the observed improvement in the overall performance of our dis-tilbert model can be presumed to extend to the model's class-level performance. ## conclusion this study proposes a novel three-class labelling system for classifying dementia in patients based on conversation transcripts. the proposed labelling system includes three classes: healthy control (hc), early stage or mild cognitive impairment (mci) and dementia. multiple models were developed utilising bert, roberta, and distil-bert. to improve the representation of dementia data, we experimented with oversampling the pitt dataset as well as combining the pitt dataset with the holland and kempler datasets to increase the number of dementia-classified data samples. the best-performing models were built upon dis-tilbert and trained on either the oversampled pitt dataset or the newly combined dataset. through hyper-parameter tuning, we achieved state-of-theart performance, including an accuracy of 98.8%, a macro-averaged f1-score of 98.6% and a precision of 100% for the dementia class. additionally, lime was employed to explain the outputs of the model and highlight the features of interest. future research could explore applying the model to more recently collected data, in line with current medical practices, to evaluate its effectiveness in real-world medical applications. furthermore, since the dementiabank database contains transcripts in multiple languages, such as german and mandarin, further research could be done to develop a multi-lingual dementia classifier to extend the benefits of these models globally.