Impact of ASR on A lzheimer’s Disease Detection: All Errors are Equal, but Deletions are More Equal than Others (2020)

Abstract: Automatic Speech Recognition (ASR) is a critical component of any fully-automated speech-based dementia detection model. However, despite years of speech recognition research, little is known about the impact of ASR accuracy on dementia detection. In this paper, we experiment with controlled amounts of artificially generated ASR errors and investigate their influence on dementia detection. We find that deletion errors affect detection performance the most, due to their impact on the features of syntactic complexity and discourse representation in speech. We show the trend to be generalisable across two different datasets for cognitive impairment detection. As a conclusion, we propose optimising the ASR to reflect a higher penalty for deletion errors in order to improve dementia detection performance.

## introduction there is a rapid growth in the number of people living with alzheimer's disease (ad) @xcite . clinical research has shown that quantifiable signs of cognitive decline associated with ad and mild cognitive impairment (mci) are detectable in spontaneous speech @xcite . machine learning (ml) models have proved to be successful in detecting ad using speech and language variables, such as syntactic and lexical complexity of language extracted from the transcripts of the speech @xcite @xcite . since transcripts should be accurate enough to properly represent syntactic and linguistic characteristics, current approaches @xcite frequently rely on 100% accurate human-created transcripts produced by trained transcriptionists. however in real-life speech-based applications of ad detection, asr is used and it produces noisy, error-prone transcripts @xcite . to our best knowledge, while the importance of well-performing asr in speech classification has been studied in depth @xcite , no prior research was done to understand what patterns of speech are influenced the most by asr errors such as word deletions and substitutions, and how this impacts performance of ad detection using ml models. in this paper, we focus on this issue and study the effect of deletion, insertion and substitution errors on lexico-syntactic language features and their resulting effect on classification performance. the effect of these errors on binary ad-healthy classification performance is studied and suggestions are provided on how to improve asr in order to maintain reasonable ad classification performance. we identify that deletion errors affect the classification more than substitution and insertion errors on two datasets of spontaneous impaired speech. the effect of these deletion errors are most profound on features related to syntactic complexity and discourse representations in speech, such as production rules, word-level structure and repetitions. these features are also identified as being the most important for the classification task using a feature gradient-based importance metric. ## changes in classification performance due to simulated errors we evaluate performance of classifying samples of speech to two classes -ad or healthy -using the db dataset. figure 1 shows that deletion errors affect classification performance significantly more than insertion and substitution errors do. 40% of deletions reduce f1 score by more than 10%, while 40% of insertions only result in 2.8%, and 40% of substitutions -in 6.3% of f1 score reduction. these differences become even more pronounced with adding a bigger amount of errors. trajectory of f1 score with varying levels of noise is substantially different from that with varying deletion errors but not that with insertions or substitutions, showing that insertion and substitution errors influence classification performance in a way that is similar to a random noise. deletion errors, however, have a significantly stronger effect on classification. it is also interesting to note that the model utilizing automatic transcripts from asr retains a level of performance at 74.96% ( ## distinctive effects of deletion errors in order to understand why deletions errors influence the classification performance significantly more than other error types, we identify features maintaining higher correlation with the amount of deletions than that with the amount of insertions and substitutions. we observe 18 features in total that distinctively correlate with deletions. out of these, the absolute majority of 15 features (83.33% of all selected) are associated with syntactic complexity (production rules of a constituency parser) and discourse phenomena (graph self-loop with 3 edges) and 3 (16.7%) -with lexical richness in speech. other feature groups, such as acoustic features or those associated with word finding difficulty, do not meet the required conditions. such results show that syntactic structure of language is much more vulnerable to deletions than to other asr errors. this can be explained by the fact that insertions and substitutions use words from the language model (i.e. most probable words) for the modifications, which to some extent helps maintain basic syntactic rules and structure. correlation between the number of deletions and features of syntactic structure shows the vulnerability of the feature group representing syntactic complexity and discourse phenomena to asr deletion errors. however, it does not explain a decrease in classification performance when adding deletion errors. in section 6 we inspect if features of syntactic complexity are more influential in ad detection than other characteristics of speech. ## model-based analysis of feature importance in order to quantify the importance of input features for classification, we obtain the gradient of the output prediction loss with respect to input features on a manually-transcribed version of the db dataset. we define gradient-based importance for feature k for an input, x i,j , in the training set for a classification model as: this importance is then averaged across the 10folds to obtain the final importance, i.e.: in order to interpret high-level patterns of input importance, we aggregate the feature importances into the groups defined in section 3.1, where aggregation of importances involves averaging the absolute gradient-importance, |imp k |, of features belonging to that group. results provided in to conclude, the feature group of syntactic complexity and discourse phenomena is affected significantly and distinctively the most by deletion errors as seen in section 5. this group is also important for classification as seen in ## generalisability evaluation in order to test how well our conclusions generalise to a different dataset of impaired speech, we repeat the same experiments performed on db on the ha dataset (section 2.1). we follow the same method, as described in section 3 to extract the features and classify samples. similarly to the results obtained on db data, with ha deletion errors affect classification performance the most. furthermore, deletion errors differentiate the same feature group of syntactic complexity and discourse phenomena: with ha dataset, 39 features correlate with deletions stronger than with insertions or substitutions, with 79.49% of features belonging to the aggregate group of syntactic complexity and discourse, and 20.51% -to the group of lexical richness. the rank of feature groups, based on the average absolute spearman correlation of all the features included in the groups, correspond to the rank observed with db dataset, with a stronger significant correlation corresponding to the group of syntactic complexity, rather than lexical richness. ## conclusions we observe that simulated deletion errors have a strong effect on classification performance when detecting cognitive impairment from speech and language, which can be traced back to their effect on syntactic complexity and discourse representations. with this observation in mind, the practical suggestion would be to optimise the asr to reflect a higher penalty for deletion errors to improve dementia detection performance. for example, the decoder can be parametrised to find a balance between insertions and deletions, so that the number of deletion errors is minimised. however, dealing with deletions in training time is not trivial, so in future work, we will focus on the optimisation of asr performance and its effect on ad detection. careful asr error management, following previous work by @xcite , could help enable strong fully-automated speechbased predictive models for dementia detection.