Multilingual Learning for Mild Cognitive Impairment Screening from a Clinical Speech Task (2021)

Abstract: The Semantic Verbal Fluency Task (SVF) is an efficient and minimally invasive speech-based screening tool for Mild Cognitive Impairment (MCI). In the SVF, testees have to produce as many words for a given semantic category as possible within 60 seconds. State-of-the-art approaches for automatic evaluation of the SVF employ word embeddings to analyze semantic similarities in these word sequences. While these approaches have proven promising in a variety of test languages, the small amount of data available for any given language limits the performance. In this paper, we for the first time investigate multilingual learning approaches for MCI classification from the SVF in order to combat data scarcity. To allow for cross-language generalisation, these approaches either rely on translation to a shared language, or make use of several distinct word embeddings. In evaluations on a multilingual corpus of older French, Dutch, and German participants (Controls=66, MCI=66), we show that our multilingual approaches clearly improve over single-language baselines.

## introduction mild cognitive impairment (mci) is a medical condition @xcite that often precedes alzheimer's disease (ad). the development of cost-effective and scaleable screening approaches for mci is crucial for the early treatment and management of ad @xcite . the semantic verbal fluency task (svf) is a promising screening approach as it combines a time-efficient testing procedure with the possibility of remote and automatic evaluation @xcite . in this task, the testee is asked to name as many words as possible from a given semantic category (e.g. animals) in a given time (e.g. 60-seconds). traditionally, the number of named within-category items is used to detect cognitive impairment. however, recent research has shown that in-depth analysis of the underlying cognitive strategies used for the svf (e.g. semantic memory retrieval, executive control) enables a more fine-grained differential diagnosis @xcite . to harness the diagnostic power of the svf, current automatic evaluation approaches identify semantic clusters in the participants' word sequences, based on semantic word embeddings @xcite @xcite . as the word embeddings used in these approaches are language-specific, training diagnostic machine learning approaches for target languages with small available datasets of svf tests is challenging. despite the potential of improving mci classification by training on larger, multilingual data, all existing approaches for automatic mci diagnosis are trained and evaluated on data from a single language. in this paper, we for the first time investigate multilingual learning approaches for mci screening from the svf. to train a joint model that generalises across test languages we evaluate two approaches: (1) translation to a common language, and (2) the application of several distinct embedding resources to the same svf productions. in line with the state of the art @xcite , we evaluate qualitative embedding-based approaches through an extrinsic quantitative downstream nlp application @xcite : classification between controls (hc) versus mci from qualitative svf features. in evaluations on french, dutch, and german corpora we show clear improvements of the multilingual learning approaches over the single-language baselines. our results show that the the performance of classical single-language, single-embedding approaches heavily depends on the combination of embedding and language, hindering generalizability. in contrast, by extracting features from several embeddings simultaneously and training over several languages, we achieve improved and more consistent classification performances across several test languages. ## related work our work is related to clinical evaluation, semantic word embeddings, as well as the automatic qualitative evaluation of verbal fluency tasks. ## results all results described in section 3.6 are in a baseline was created using single language, single embedding (sl-se) classifications. in the untranslated approach, dutch had the lowest average across embeddings with an average auc of 0.29, then french with 0.58, and finally german with 0.55. similarly, dutch had the lowest value (average @xmath0.38) across resources in the in the case of the untranslated data, we see the largest overall improvement in the multilingual, multi-embedding scenario. averaging over the cross language averages (lang avg) of the ml-se scenario produces an auc of 0.60, while the ml-me average reaches an auc of 0.66. for the ml-me scenrios in the translated case, by averaging across the ml-se models we achieve a comparable auc of 0.58 which is outperformed by the ml-me model (0.62). however, in the translated case, the best results are produced in the multilingual scenario using wiki2vec embeddings (0.64). in addition to the classification analysis, we investigated the out-of-vocabulary (oov) rate for each of the embeddings in each of the languages as a form of quality control. the results in ## discussion in this study we investigate two different methods of combining multilingual data to build clinical models to distinguish between healthy controls and early signs of alzheiemer's disease (mci); untranslated and translated. while the multi-embedding method is best for when data is kept multilingual, if the data is translated and no longer in need of multilingual re-sources, a single embedding type did emerge with the best performance, wiki2vec. given this, in the case where a common language (especially english) can be achieved, according to our data, it may be best to find and use one embedding type. in addition, we found that embedding type does make a difference in classification performance. therefore, caution should be used when deciding on semantic resources. for instance, in the untranslated case, if we were to build a multilingual model and only use spacy emebeddings, we would have relatively good performing classifier in french and german but the dutch classification would not exceed chance performance. while combining embeddings may not yield the best results for each individual language, it results in the most uniform improvement in a multilingual-versus translatedsetting. however, translating the data to english, drastically improved dutch performance, specifically with the wiki2vec models. speculatively, the improved overall performance of the english translations with wiki2vec could be due to the backlink model where relationships are modeled through linked wikipedia page, situating wiki2vec to be very useful in modelling these semantic relationship from a cognitive standpoint in verbal fluency tasks. however, based on these results, it seems that these relationships are mainly found in the english wiki2vec model, most likely due to the large discrepancy in the amount of available training data between the languages. beyond just the brand of embedding, there are pros and cons that come with each the untranslated and translated approach. by translating the data to english, we introduce possible errors based on how the data is translated. in this study, we chose to combine an automatic approach with a manual post-editing step, making the translated approach not fully automatic. from a clinical perspective, we do not know if the previous work on cognition applies to data that is translated to another language and then assessed. however, from a computational standpoint, if a reliable translation service for the source language to english exists, using the monolith of english resources presents as a reliable and effective alternative. there are many challenges that arise when trying to concatenate data from multiple sources, thus specific caution should be taken on how to model data that has health implications. our investigation of the two approaches (untranslated and translated) shows that svf speech data can be combined to achieve results comparable to previous models. as no unified benchmark exists for hc vs. mci detection from the svf, our results can only cautiously be compared with previous work. however, we noticed that our best results from our best models for french(0.66), german(0.68) and dutch (0.69) are in line with reported auc values for french (0.76 @xcite ) and spanish (0.75 @xcite ). it is worth noting, that these results are achieved without using the overall word count, which typically the strongest indicator for mci detection from the svf task. ## conclusion using multilingual cognitive data in both a untranslated multilingual, multi-embedding approach and translated to a common language approach improved classification over single language baselines. this is promising not only for the feasibility of increasing the size of small clinical datasets in quick and cost-effective way, but it also opens the door for methodology on how we can use multilingual data to build more robust understanding of underlying cognitive conditions @xcite . future work should look at exploratory analysis for the compatibility of features computed from translated transcripts in the current clinical understanding. this could present translation as viable option for low-resource languages, or taking advantage of larger resources, while still presenting explainable clinical solutions and improved classification performance.while the languages in this study are all in the same language family, this methodology should be tested with data from different language families to test for robustness of the solution. as such, our proposed methodology provides insight into the effect of nlp resources for classifications on cognition as well as a tentative solution to the problem of combining multiple clinical datasets. this addresses the issue of small clinical data sets as well as opens the door for building robust models of cognition for clinically actionable solutions leveraging multilingual data, paving the way towards reaching the societal goal of cost-effective early ad detection.