Text Classification by Contrastive Learning and Cross-lingual Data Augmentation for A lzheimer’s Disease Detection (2020)

Abstract: Data scarcity is always a constraint on analyzing speech transcriptions for automatic Alzheimer’s disease (AD) detection, especially when the subjects are non-English speakers. To deal with this issue, this paper first proposes a contrastive learning method to obtain effective representations for text classification based on monolingual embeddings of BERT. Furthermore, a cross-lingual data augmentation method is designed by building autoencoders to learn the text representations shared by both languages. Experiments on a Mandarin AD corpus show that the contrastive learning method can achieve better detection accuracy than conventional CNN-based and BERTbased methods. Our cross-lingual data augmentation method also outperforms other compared methods when using another English AD corpus for augmentation. Finally, a best detection accuracy of 81.6% is obtained by our proposed methods on the Mandarin AD corpus.

## introduction alzheimer's disease (ad) is a worldwide popular neurodegenerative disease and the major cause of dementia. it is estimated that 32.8 million people worldwide living with ad in 2015, resulting in economic costs of 604 billion us dollars, and the number of patients is expected to almost double every 20 years @xcite . with no cure has been found yet, in order to allow effective intervention in early stages to delay onset of the disease, early detection of ad is crucial @xcite . nonetheless, nowadays common methods of ad diagnosis, including cognitive tests and biomarker detection, are too time-consuming and expensive to cover all potential patients. thus, developing an effective and convenient method for early ad detection becomes a valuable research topic. language impairment generally appears at the early stages of ad @xcite . suffering the disease, patients have significant worse performance than controls on a standard aphasia test, specifically in the areas of verbal expression, auditory comprehension, repetition, reading, and writing @xcite . the performances of ad patients on some language tasks, such as picture description and sentence repetition, are distinctive from healthy people, which give us a hint to utilize these tasks to detect ad. other advantages of these language tasks include their simple procedure and quick process. one usually completes the picture description task within 5 minutes and needs few instructions. a lot of researches have applied machine learning approaches to automatic ad detection based on language tasks. @xcite extracted total 370 features considering part-of-speech, syntax, acoustics, as well as other aspects of linguistics, and obtained a best average accuracy of 82% for binary classification using the 35 top-ranked features on the dementiabank corpus. @xcite designed a single feature by building two trigram language models from the speech transcriptions of ad patients and controls respectively, and resulted in an eer of 77%. several later studies @xcite @xcite continued to show the effectiveness of language model features. ad detection based on speech transcriptions is essentially a text classification task. various models have been proposed to solve general text classification tasks, such as convolutional neural networks @xcite , recurrent neural networks @xcite , and hierarchical attention networks @xcite . recently, the advent of large-scale self-supervised pretraining has played a central role in the progress of natural language processing (nlp) research @xcite @xcite . in particular, bert @xcite greatly improved the performance of several general nlp benchmarks such as glue @xcite and superglue @xcite . fine-tuning bert is a straightforward and powerful solution to text classification. however, in the ad detection case, it faces a major challenge, which is data scarcity. data scarcity is always an issue of building automatic ad detection models, and the difficulties in recruiting eligible subjects make it hard to overcome. it constrains model capacity as powerful models usually have a large number of trainable parameters and tend to overfit small datasets. the dementiabank corpus @xcite contains only 256 cases labeled as "possible" or "probable" ad, which is still larger than most of the similar datasets in other language, such as japanese @xcite , portuguese @xcite , and mandarin chinese @xcite . the mandarin corpus used in this study contains 104 samples of ad patients, which is also much smaller than dementiabank. in order to deal with the issue of data scarcity, this paper designs a contrastive learning method to learn effective representations for text classification. first, we rely on bert extracting contextual word embeddings for every sample in the corpus. then, a weighted sum of word embeddings, i.e. a paragraph vector, is obtained by contrastive learning and finally a similarity-based decision is made for classifying test samples. since embedding features of samples are fixed as input and there are no model parameters in the similarity-based decision, this method is expected to ease the problem of data scarcity. additionally, an autoencoder-based cross-lingual data augmentation method is proposed to augment the mandarin corpus with a larger english one. since merging the bert-based paragraph vectors of both languages directly to build a classifier is unsuitable, we train an autoencoder-based model to obtain the latent representations shared by both languages, where a large general-domain parallel corpus is utilized. then, the encoded paragraph vectors of both languages can be combined to estimate the classifier for mandarin transcriptions. our contributions in this paper are twofold. first, a contrastive learning method is designed which studies effective representations for ad detection based on bert embeddings. experimental results show that this method achieves better detection accuracy than conventional cnn-based and bert-based methods by 3.9% at least on our mandarin corpus. second, an autoencoder-based method is further proposed to augment original mandarin corpus with a larger english one, which outperforms other compared data augmentation methods by 1.4% at least in our experiments. ## datasets the mandarin corpus used in this paper was collected at the shanghai tongji hospital @xcite . all participants were with the complaint of memory impairment and underwent a comprehensive neuropsychological battery, including the mini-mental state examination @xcite . diagnosis was made according to the core clinical criteria to dementia of nia-aa established in 2011 @xcite . the corpus contains the transcriptions of the mandarin speech from 208 participants on the cookie theft picture description task, 104 of them were healthy controls and the other 104 were diagnosed as ad patients. this task asked participants to say whatever happened in the cookie theft picture as much as possible, and allowed encouragement from interviewers when participants had difficulties. speech recordings were manually transcribed following the chat (codes for the human analysis of transcripts) protocol @xcite , same as dementiabank. the average length of dialogue samples in the this corpus is 127.4 characters with a standard deviation of 61.7 characters. demographics of participants in the corpus are shown in the english dementiabank (db) corpus was used for data augmentation in our experiments. the db corpus contains 498 samples from 292 participants taking the cookie theft picture description task, 242 of these samples were labeled as healthy controls and the other 256 were labeled as "possible" or "probable" ad, which we refer as an ad group. each sample has an audio recording together with its corresponding transcription. only the transcriptions are used in this study. the average length of dialogue samples in the corpus is 122.7 words with a standard deviation is 66.9 words. demographics of participants in this corpus are also shown in in our proposed cross-lingual data augmentation method, a parallel mandarin-english corpus is necessary to train the autoencoders for deriving the paragraph representations shared by both languages. the opensubtitle corpus (lison and tiedemann, 2016) was adopted as the parallel corpus in our implementation. we utilized the mandarin-english subset of this corpus, which contains 9.9 million lines of dialogue. ## conclusions this paper first presents a contrastive learning method to study effective text representations from small corpus for ad detection. this method contains a bert feature extractor and a trainable pooling layer. it adopts similarity-based decision, which performed better than building separate classifiers in our experiments on a mandarin ad corpus. an autoencoder-based cross-lingual data augmentation method is further proposed for utilizing another english ad corpus to improve model performance on the mandarin one. this method outperformed other cross-lingual data augmentation methods, achieving a best accuracy of 81.6% in our experiments. exploring more effective pretrained models and better methods for learning bilingual text representations will be the task of our future work.