Towards Domain-Agnostic and Domain-Adaptive Dementia Detection from Spoken Language (2023)

Abstract: Health-related speech datasets are often small and varied in focus. This makes it difficult to leverage them to effectively support healthcare goals. Robust transfer of linguistic features across different datasets orbiting the same goal carries potential to address this concern. To test this hypothesis, we experiment with domain adaptation (DA) techniques on heterogeneous spoken language data to evaluate generalizability across diverse datasets for a common task: dementia detection. We find that adapted models exhibit better performance across conversational and task-oriented datasets. The feature-augmented DA method achieves a 22% increase in accuracy adapting from a conversational to task-specific dataset compared to a jointly trained baseline. This suggests promising capacity of these techniques to allow for productive use of disparate data for a complex spoken language healthcare task.

## introduction data-driven models of diverse conditions affecting spoken language abilities offer promising realworld benefits @xcite . however, the datasets available for developing these models are often small and disparate, spanning varied diagnostic and non-diagnostic tasks mapped to different taxonomies at conflicting granularities @xcite . this has largely constrained progress to models excelling in specialized settings (e.g., individuals with homogeneous language background describing a standardized image @xcite ). at the same time, it has created challenges in building more generalizable knowledge about language patterns associated with the condition of interest @xcite . outside healthcare applications, domain adaptation (da) has long been applied to increase the capacity of nlp systems to leverage meaningful information from diverse data @xcite . these techniques generally seek to harness data from one domain (the source) to improve performance in another (the target). usually the target domain has little or no labeled data, while the source has a relatively large amount of labeled data. despite the advantages offered by da for many nlp problems, it has remained under-studied for healthcare tasks due to numerous complexities of healthcare data @xcite . nonetheless, most healthcare problems offer the ideal learning settings in which da is designed to thrive. we present a systematic analysis of the use of da for a low-resource healthcare problem that has recently been popular in the nlp community: dementia. we adopt a wide definition of dementia in our work, encompassing datasets pertaining to alzheimer's disease or related dementia (adrd) and age-related mild cognitive impairment (mci), in line with current nlp community norms @xcite . our research questions include: q1. can da be used to exploit spoken language data pertaining to dementia from one domain, to improve its detection in other domains? q2. if yes, does this offer performance improvements over simpler joint training? q3. how do different linguistic features and class biases contribute to this performance? we define domain in this study as a distinct dataset with supervised labels describing dementia status in some capacity. data collection protocol and precise labeling taxonomy may vary across domains, making our task slightly more complex than related work that focused solely on differences in source language @xcite or labeling taxonomy @xcite . we find that da can indeed support improved dementia detection across domains compared to joint training, and we identify key pivot features and factors contributing to this success. it is our hope that continued study of da in healthcare applications can further extend the boundaries of our understanding and promote impactful follow-up work. ## related work most prior work on developing spoken language models of dementia has followed a common pattern, focusing on designing and evaluating datasetspecific approaches. this has included (most popularly) a picture description task @xcite @xcite , as well as other datasets with more open-ended conversational speech @xcite @xcite . these models singularly focus on the source domain, with no expectation of deployment beyond that, opening questions about their ability to generalize beyond small, publicly available reference sets. the extent to which da has been explored in this context is limited. @xcite leveraged transfer learning, one form of da that involves fine-tuning a model pretrained on a much larger dataset using the smaller target domain dataset, to study the perplexity ratio of normal and artificially degraded transformer-based language models for dementia detection. likewise, @xcite achieved performance boosts in detecting early signs of aphasia in cross-language settings compared to the unilingual baseline using optimal transport domain adaptation. a problem with transfer learning in many healthcare contexts is that target datasets are much smaller than for other nlp tasks for which the technique has demonstrated success. the benefits of transfer learning do not necessarily transfer (no pun intended) to ultra lowresource settings, where resulting models may be much less stable @xcite . other forms of da that may be more suited to dementia detection and other very low-resource healthcare problems are feature-based and instancebased da. both were originally leveraged for smaller datasets closer in scale to (although still larger than) those available for dementia detection @xcite , making it a promising and perhaps under-appreciated alternative to transfer learning. feature-based da focuses on modifying the feature space of the source and target datasets in some way that promotes the classifier's ability to generalize across them. @xcite experimented with two feature-based da techniques to adapt separate domain subsets split from the same source dataset, dementiabank @xcite . instance-based da focuses on reweighting instances based on their importance to the target domain task @xcite . it has not yet been studied for dementia detection. we build upon @xcite 's promising findings by studying the effects of numerous feature-based and instance-based da techniques across different dementia datasets with conversational and task-related speech samples. ## analysis the results in tables 3-6 clearly answer our first research question (q1), demonstrating that da can be used to exploit spoken language data pertaining to dementia detection in one domain to improve its detection in other domains. they also answer q2, showing that da offers performance improvements over jointly training on data from multiple domains. to answer q3, we performed additional analyses to probe the contributions of feature subsets and class bias to overall performance. ## discussion and conclusions our work reveals intriguing findings on the use of da for dementia detection. first, we find that da can be successfully leveraged to improve featurebased dementia detection performance. this is the most comprehensive study of feature-based da for this task, and the first to consider instance-based da. we find that feature-based da outperforms instance-based da, and that an approach allowing for multiple source domains (multiaugment) holds promise in many cases. in general, f 1 score is similar across target datasets, ranging from 0.76 (ccc) to 0.77 (db and adress). our da conditions also exhibit clear performance improvements over jointly training on the same data, offering further evidence to support the use of da for this task. finally, in follow-up studies on the importance of individual features and class biases in this setting, we find that pivot features pertaining to number of tokens, average phrase length, acoustic qualities, named entities, and measures of vocabulary richness and lexical frequency are particularly critical to strong performance. this suggests that these features may be particularly robust across domains. we also demonstrate that the performance of da conditions relative to joint training is not due to domain-specific class bias, further strengthening our conclusions. in the future, we hope to conduct follow-up studies to further probe the limits and nuances of da applied to this and other low-resource healthcare tasks.