Robustness and Sensitivity of BERT Models Predicting A lzheimer’s Disease from Text (2021)

Abstract: Understanding robustness and sensitivity of BERT models predicting Alzheimer’s disease from text is important for both developing better classification models and for understanding their capabilities and limitations. In this paper, we analyze how a controlled amount of desired and undesired text alterations impacts performance of BERT. We show that BERT is robust to natural linguistic variations in text. On the other hand, we show that BERT is not sensitive to removing clinically important information from text.

## introduction alzheimer's disease (ad) is a prevalent neurodegerative condition that inhibits cognitive abilities and impacts one's language abilities. for example, cognitively impaired people tend to use more pronouns instead of nouns, and pause more often between sentences in narrative speech @xcite . this insight makes automatic detection possible. machine learning (ml) classifiers can detect cognitive impairments given descriptive linguistic features or using pre-trained large language models @xcite . bert is a model that achieves promising performance on a variety of tasks, including ad prediction from speech and language @xcite . however, this promising performance may be fallacious, i.e. deep neural language models may learn pseudo patterns from training data to attain high performance on test sets @xcite @xcite @xcite . therefore, in order to be confident in the outcomes of bert models classifying ad it is important to assess whether these models are robust to some natural noise that may be introduced in language. it is also important to know if bert models are sensitive to the aspects that are considered to be important for recognizing cognitive impairment from human language. in this paper, we analyze robustness and sensitivity of bert models in their ability to classify ad from text by analysing the effect of noise, introduced from artificial text perturbations, on the performance of the model. some previous research was conducted on the impact of asr-related noise on dementia detection @xcite , as well as the effect of artificial text alterations on ad classification @xcite . however, these previous studies only focus on conventional classification models, such as random forest and svm. to the best of our knowledge, we are the first to analyse how the noise introduced by texts perturbations impact bert models, in the domain of ad classification. ## results the results of testing the fine-tuned bert model on the variety of perturbed versions of the adress test set show that the performance changes differently depending on different types of text alterations ( ## discussion type of perturbation correlation between w 1 and acc f1 prec rec spec deleting informational units 0.24 0.56 -0.07 0.52 -0.21 substituting with the most similar word (via word2vec embeddings) 0.77 0.89 0.89 0.87 -0.91 deep representation (vgg-16) -0.26 -0.37 0.10 -0.45 0.41 interestingly, recall and specificity values seem to show the opposite results here -specificity decreases by up to 21%, while recall stays on the same level or even increases by 4-9%. substituting words with their similar alternatives or synonyms may be understood as increasing the level of lexical complexity, i.e. the model is introduced with multiple, maybe less usual, ways to express the same meaning. it is known that lexical complexity is one of the prominent ways that allow detecting cognitive impairment from language. thus such an implicit way to change the lexical complexity of texts seems to help the bert model in reducing the amount of true positive errors while detecting ad. however, more than 40% of such substitutions may make the original texts less realistic, which, as we see from the results, substantially reduces model performance, including reducing recall level. desired change: other types of text alterations, such as removal of information units or tokens representing filled pauses, are not considered to be natural noise. as these characteristics of language are clinically important in detecting cognitive impairment, the models should be sensitive to such changes in language. our results show that the finetuned bert model ignores completely removal of filled pauses. performance of the model decreases by 3-5% of f1 as a reaction to deleting different types of information units but this change is not significant (mcnemar's test @xmath0.18). this change in performance is similar to the change caused by synonym substitution and shows that the model is not sensitive enough to removal of clinically relevant information. this leads us to inspect how each type of alterations affects distributional shift from the original text and whether there is a relation between the shift and model's performance. ## limitations and conclusions in this work, we analysed how the controlled amount of desired and undesired text alterations impacts bert classification performance in the domain of ad detection. we showed that bert is robust enough to the natural linguistic noise, although the model is biased towards text samples of non-ad females. on the other hand, bert is not sensitive enough to removal of clinically relevant information. this lack of sensitivity is not directly influenced by distributional shift. this work is a first step towards investigating bert models' robustness and sensitivity in the domain of ad detection from text, and we only report empirical results of one bert model fine-tuned and tested on one dataset. more work should be done in this area to ensure the results are widely generalizable within the domain. textual data used in our experiment represent transcribed conversational speech and as such, may be quite different from other types of texts, e.g. written text. future work is necessary to see if the effect of text alterations remain the same with other types of text.