A Tale of Two Perplexities: Sensitivity of Neural Language Models to Lexical Retrieval Deficits in Dementia of the A lzheimer’s Type (2020)

Abstract: In recent years there has been a burgeoning interest in the use of computational methods to distinguish between elicited speech samples produced by patients with dementia, and those from healthy controls. The difference between perplexity estimates from two neural language models (LMs) - one trained on transcripts of speech produced by healthy participants and one trained on those with dementia - as a single feature for diagnostic classification of unseen transcripts has been shown to produce state-of-the-art performance. However, little is known about why this approach is effective, and on account of the lack of case/control matching in the most widely-used evaluation set of transcripts (DementiaBank), it is unclear if these approaches are truly diagnostic, or are sensitive to other variables. In this paper, we interrogate neural LMs trained on participants with and without dementia by using synthetic narratives previously developed to simulate progressive semantic dementia by manipulating lexical frequency. We find that perplexity of neural LMs is strongly and differentially associated with lexical frequency, and that using a mixture model resulting from interpolating control and dementia LMs improves upon the current state-of-the-art for models trained on transcript text exclusively.

## introduction alzheimer's disease (ad) is a debilitating neurodegenerative condition which currently has no cure, and dementia of the alzheimer's type (dat) is one of the most prominent manifestations of ad pathology. prior to availability of diseasemodifying therapies, it is important to focus on reducing the emotional and financial burden of this devastating disease on patients, caregivers, and the healthcare system. recent longitudinal studies of ⇤ denotes equal contribution aging show that cognitive manifestations of future dementia may appear as early as 18 years prior to clinical diagnosis -much earlier than previously believed @xcite . with 30-40% of healthy adults subjectively reporting forgetfulness on a regular basis @xcite , there is an urgent need to develop sensitive and specific, easy-to-use, safe, and costeffective tools for monitoring ad-specific cognitive markers in individuals concerned about their cognitive function. lack of clear diagnosis and prognosis, possibly for an extended period of time (i.e., many years), in this situation can produce uncertainty and negatively impact planning of future care @xcite , and misattribution of ad symptoms to personality changes can lead to family conflict and social isolation @xcite . delayed diagnosis also results in an estimated $7.9 trillion in medical and care costs @xcite due to high utilization of emergency care, amongst other factors, by patients with undiagnosed ad. cognitive status is reflected in spoken language. as manual analysis of such data is prohibitively time-consuming, the development and evaluation of computational methods through which symptoms of ad and other dementias can be identified on the basis of linguistic anomalies observed in transcripts of elicited speech samples have intensified in the last several years @xcite @xcite . this work has generally employed a supervised machine learning paradigm, in which a model is trained to distinguish between speech samples produced by patients with dementia and those from controls, using a set of deliberately engineered or computationally identified features. however, on account of the limited training data available, overfitting is a concern. this is particularly problematic in dat, where the nature of linguistic anomalies varies between patients, and with ad progression @xcite . in the current study we take a different approach, focusing our attention on the perplexity of a speech sample as estimated by neural lms trained on transcripts of the speech of participants completing a cognitive task. to date, the most successful approach to using lm perplexity as a sole distinguishing feature between narratives by dementia patients and controls was proposed by @xcite and replicated by @xcite . the approach consists of training two recurrent neural lms -one on transcripts from patients with dementia and the other on transcripts from controls. the difference between the perplexities estimated with these two lms results in very high classification accuracy (auc: 0.92) reported by both studies. the explanation for this performance offered by @xcite relies on observations that patients with dat describe the picture in an unforeseen way and their speech frequently diverts from the content of the picture, contains repetitions, incomplete utterances, and refers to objects in the picture using words like "thing" or "something". this explanation, however, conflicts with the findings by @xcite that demonstrate similarly high classification accuracy (auc: 0.91) with a single hidden layer non-recurrent neural network and bag-of-words input features, suggesting that while word sequences play a role, it may not be as large as previously believed by @xcite . @xcite explanation contrasts "local" with "global language properties" of the picture descriptions being captured by recurrent neural lms vs. the non-recurrent bag-of-words neural network classifier, respectively. both of these explanations are based on informal qualitative observations of the data and are not entirely satisfying because both fail to explain the fact that it is precisely the difference between the control and dementia lms that is able to discriminate between patients and controls. the individual lms are not nearly as good at this categorization task. the objective of the current study is to quantify the extent to which the differences between neural lms trained on language produced by dat patients and controls reflect known deficits in language use in this disease -in particular the loss of access to relatively infrequent terms that occurs with disease progression @xcite . we approach this objective by interrogating trained neural lms with two methods: interrogation by perturbation in which we evaluate how trained neural lms respond to text that has been deliberately perturbed to simulate ad progression; and interrogation by interpolation in which we develop and evaluate hybrid lms by interpolating between neural lms modeling language use with and without dementia. we find neural lms are progressively more perplexed by text simulating disease of greater severity, and that this perplexity decreases with increasing contributions of a lm trained on transcripts from patients with ad, but increases again when only this lm is considered. motivated by these observations, we modify the approach of @xcite by incorporating an interpolated model and pre-trained word embeddings, with improvements in performance over the best results reported for models trained on transcript text exclusively. ## results and discussion figure 1 : relationship between log frequency bands used to replace words in synthetic cookie theft picture descriptions to simulate degrees of semantic dementia and perplexity of lstm language models trained on picture descriptions by controls and dementia patients. lstm model has equivalent performance to the rwthlm lstm model. having replicated results of previously published studies and confirmed that using the difference in perplexities trained on narratives by controls and dementia patients is indeed the current state-of-theart, we now turn to explaining why the difference between these lms is much more successful than the individual models alone. first, we used the six "cookie theft" narratives designed to simulate semantic dementia to examine the relationship between p con and p dem with gluonnlp lstm lms and log lexical frequency bands. the results of this analysis are illustrated in figure 1 and show that p dem is higher than p con on narratives in the lower log frequency bands (less simulated impairment) and lower in the higher log frequency bands (more simulated impairment). we confirmed these results by calculating mean log lexical frequency on all dementiabank narratives and fitting a linear regression model to test for associations with perplexities of the two lms. the regression model contained mean lexical frequency as the dependent variable and p dem and p con as independent variables, adjusted for age, education and the length of the picture description narrative. in order to avoid likely practice effects across multiple transcripts, we only used the transcript obtained on the initial baseline visit; however, we did confirm these results by using all transcripts to fit mixed effects models with random slopes and intercepts in order to account for the correlation between transcripts from the same subject (mixed effects modeling results not shown). the results demonstrate that the association between perplexity and lexical frequency is significant and positive for the control lm (coeff: 0.563, @xmath0.001) and negative for dementia lm (coeff: -0.543, @xmath1.001). age, years of education, and length of the narrative were not significantly associated with lexical frequency in this model. these associations show that the control lm and dementia lm are more "surprised" by narratives containing words of higher lexical frequency and lower lexical frequency respectively. if the use of higher lexical frequency items on a picture description task portends a semantic deficit, then this particular pattern of results explains why it is the difference between the two models that is most sensitive to manifestations of dementia and suggests that there is a point at which the two models become equally "surprised" with a difference between their perplexities close to zero. in figure 1 , that point is between log lexical frequency bands of 2.0 and 2.5 corresponding to the mild to moderate degree of semantic impairment reported by @xcite . notably, in the clinical setting, the mild forms of dementia such as mild cognitive impairment and mild dementia are also particularly challenging and require integration of multiple sources of evidence for accurate diagnosis @xcite . a practical issue for automated methods to detect dementia concerns establishing their accuracy at earlier stages of disease progression, where a readily disseminable screening tool would arguably have greatest clinical utility, especially in the presence of an effective disease-modifying therapy. to this end, @xcite defined a "screening scenario" in which evaluation was limited to participants with a last available mmse of 21 or more, which corresponds to a range of severity encompassing mild, questionable or absent dementia @xcite . in this scenario, classification accuracy of the 'paired perplexity' lstm based model was only slightly lower (auc: 0.87) than the accuracy on the full range of cognitive impairment (auc: 0.92). we found similar performance with our models. when limiting evaluation to those participants with a last-recorded mmse 21, average aucs across 10 loocv iterations were 0.836 ±0.014, 0.879 ±0.01, 0.893 ±0.004, and 0.899 ±0.012 for the baseline @xcite ), pretrained embeddings, interpolated (↵=0.75) and interpolated (↵=0.75) with pretrained embeddings variants, respectively. these results support the notion that paired neural lms can be used effectively to screen for possible dementia at earlier stages of cognitive impairment. the contributions of our work can be summarized as follows. first, our results demonstrate that the relationship between lm perplexity and lexical frequency is consistent with the phenomenology of dat and its deleterious effects on patients' vocabulary. we show that the "two perplexities" approach is successful at distinguishing between cases and controls in the dementiabank corpus because of its ability to capture specifically linguistic manifestations of the disease. second, we observe that interpolating between dementia and control lms mitigates the tendency of dementia-based lms to be "surprised" by transcripts indicating severe dementia, which is detrimental to performance when the difference between these lms is used as a basis for classification. in addition, we find a similar smoothing effect when using pre-trained word embeddings in place of a randomly instantiated word embedding layer. finally, we develop a modification of fritsch et al's "two perplexity" approach that is consistent with these observations -replacing the dementia model with an interpolated variant, and introducing pre-trained word embeddings at the embedding layer. both modifications exhibit significant improvements in performance, with best results obtained by using them in tandem. though not strictly comparable on account of differences in segmentation of the corpus amongst others, we note the performance obtained also exceeds that reported with models trained on text alone in prior research. code to reproduce the results of our experiments is available on github foot_10 . while using transcript text directly is appealing in its simplicity, others have reported substantial improvements in performance when pos tags and paralinguistic features are incorporated, suggesting fruitful directions for future research. furthermore, prior work on using acoustic features shows that they can contribute to discriminative models @xcite ; however, dementia bank audio is challenging for acoustic analysis due to poor quality and background noise. lastly, while our results do support the claim that classification occurs on the basis of dementia-specific linguistic anomalies, we also acknowledge that dementia-bank remains a relatively small corpus by machine learning standards, and that more robust validation would require additional datasets. ## conclusion we offer an empirical explanation for the success of the difference between neural lm perplexities in discriminating between dat patients and controls, involving lexical frequency effects. interrogation of control-and dementia-based lms using synthetic transcripts and interpolation of parameters reveals inconsistencies harmful to model performance that can be remediated by incorporating interpolated models and pre-trained embeddings, with significant performance improvements.