Rare-Class Dialogue Act Tagging for A lzheimer’s Disease Diagnosis (2021)

Abstract: Alzheimer’s Disease (AD) is associated with many characteristic changes, not only in an individual’s language but also in the interactive patterns observed in dialogue. The most indicative changes of this latter kind tend to be associated with relatively rare dialogue acts (DAs), such as those involved in clarification exchanges and responses to particular kinds of questions. However, most existing work in DA tagging focuses on improving average performance, effectively prioritizing more frequent classes; it thus gives a poor performance on these rarer classes and is not suited for application to AD analysis. In this paper, we investigate tagging specifically for rare class DAs, using a hierarchical BiLSTM model with various ways of incorporating information from previous utterances and DA tags in context. We show that this can give good performance for rare DA classes on both the general Switchboard corpus (SwDA) and an AD-specific conversational dataset, the Carolinas Conversation Collection (CCC); and that the tagger outputs then contribute useful information for distinguishing patients with and without AD

## introduction natural language processing (nlp) has been applied to clinical health data for many purposes, including summarizing clinical notes, extracting specific elements from an unstructured medical record, and question-answer systems to interact with patients @xcite @xcite . within this, one recent focus is on the use of nlp to diagnose the presence or extent of neurodegenerative cognitive impairment and/or monitor changes, based on patients' speech and language (see e.g. @xcite , with much of this work focussing on dementia, primarily alzheimer's disease (ad) (see e.g. @xcite . most such approaches are based on features of the speaker's (or writer's) individual language, e.g. the complexity of vocabulary or syntax (see e.g. @xcite , for a comparison of a range of such features). however, conditions such as ad also affect communication in interaction: ad patients display more conversational problems, often use terms that signal misunderstanding, and produce more requests for repair; while their conversational partners produce more elaboration or clarification (see e.g. @xcite . closed (yes/no) questions are also asked more frequently of ad patients than open-ended wh-questions @xcite , and patients' ability to respond can vary with question type @xcite . differences in dialogue act (da) profiles might therefore add useful information for automatic diagnosis and monitoring of ad, and might also generalise better across languages than more lexically-or syntactically-based approaches: clarification and non-understanding signals seem to be quite general across languages and cultures @xcite . however, while some computational studies have used interactional differences in ad diagnosis (see e.g. @xcite , these use models which are not interpretable in these da terms, making it hard to provide useful output to clinical researchers, clinicians or carers. here, we therefore apply an explicit da tagging approach to the problem, specifically looking for das that are characteristic of dementia, e.g. signals of non-understanding, requests for clarification, and particular types of questions and answers. many of these are rare in natural dialogue, though; the signal non-understanding da, for example, makes up only 0.1% of utterances in the switchboard corpus @xcite . standard da tagging approaches, trained on average loss across all da classes, therefore fail to give good performance. the main contributions of this paper are as follows: interaction and ad diagnosis as explained above, ad patients display a number of characteristic interaction differences which can be characterised in terms of dialogue acts (das), including the rate of misunderstanding or non-understanding signals, requests for repair, elaboration, and clarification @xcite , as well as yes/no-questions, wh-questions and choice questions and responses thereto @xcite @xcite . however, these studies, often based on conversation analysis (ca), give rich detail but are small-scale and/or qualitative. some more quantitative corpus-based work makes similar observations: nasreen et al. (2019) examine da distributions in the carolinas conversation collection (ccc, pope and davis, 2011), finding more signal-non-understanding, simple yesanswers and clarification requests in cognitively impaired patients' conversations. computational work that leverages these features is rare, however. many diagnosis classification models include some signals associated with non-understanding (e.g. @xcite but only as part of large general language feature sets. one reason for this is that many studies use data that contains little interaction: the commonly used dementiabank pitt corpus, for example, contains conversations of a very one-sided nature. in a recent study, @xcite developed an annotation scheme with 26 das based on iso standard @xcite on dementiabank data set to facilitate automated cognitive health screening from conversational interviews. they investigated phenomena like clarification request but some of the tags are specific to cookie theft picture description task @xcite and are not very general. some recent work uses a more truly interactive approach: @xcite use a probabilistic graphical model to classify ad patients in the ccc corpus, although they use pauses and vocalisation times rather than any da information; @xcite include interactional features in a svm classifier on @xcite 's dataset, showing good accuracy, but use very specific features (e.g. "responding to neurologists' questions about memory problems") rather than more general da tags. in contrast, our goal here is to investigate the use of general, well-known (but rare) da classes. dialogue act (da) tagging da tagging has been approached using a range of machine learning techniques, starting with early work using hidden markov models to capture the intuition that key information lies in both the sequences of words within utterances and the sequence of das across utterances @xcite . improvements have been gained by using conditional random fields @xcite , cue phrase models @xcite , joint classification and segmentation @xcite , and more recently neural networks including recurrent neural networks (rnns) @xcite and convolutional neural networks (cnns) @xcite . most recent work sticks with @xcite 's original intuition to include contextual information (preceding utterances and their da roles help predict the current utterance), often via hierarchical models where the higher layers capture da/utterance sequence information; see e.g. @xcite 's use of a crf above dialogue-level and utterance-level bilstms, achieving state-of-theart accuracy of 82.9% on the standard swda corpus. however, variants exist: @xcite , for example, consider only a limited number of preceding utterances as a context within a rnn, rather than the full sequence, accuracy is reduced to 77.34% on swda but their model, in using only limited preceding context (rather than assuming knowledge of future utterances) is suitable for incremental online settings. rare da classes all these approaches, however, train and evaluate their models assuming that the goal is average performance over a general da tagset, usually the 42-tag swda damsl scheme @xcite . some use fewer classes - fuscone et al. (2020) 2009 ) group the 42 classes into 7 common classes and one 'other' category based on frequency -but all of these focus on the most common tags. in contrast, we are interested in the rare classes useful for dementia analysis, following the clinical ca work described above; we give a full list of these classes of interest in section 4.1 (see ## proposed approach here, then, our purpose is to improve da tagging accuracy for the specific da classes of interest in ad diagnosis, including specific types of questions, answers and misunderstanding signals, most of which are relatively rare. for this purpose, we use a context-based hierarchical bilstm model with attention, to capture relations at the word, utterance and da level and leverage utterance da/context information. to maintain the ability to use our model in an online setting, we use only utterances from the preceding (left) context, not the following (right) context. we perform da tagging experiments on two corpora, one general and one ad-specific, to compare a range of models: ## experiments 4.1 da filtering to keep our approach as domain-and datasetgeneral as possible, we start with the standard damsl tagset @xcite and adapt it. based on the clinical studies described in section 2, we keep 17 specific da tags of interest from damsl; split 2 of them each into 2 sub-categories; and collapse all other tags into a single other tag, giving a total of 20 tags. the two new da tags are clarification-request (qc) and statementanswer (sa): clarification-request (qc) is a subcategory of signal-non-understanding (br) which requests more specific information (see e.g. @xcite ; while statement-answer (sa) is a sub-category of declarative-statement (sd) used as an answer to a wh-question (qw), open-question (qo) or orquestion (qr). the full tagset foot_0 is shown in ## conclusion this work has presented a da tagger (a hierarchical bilstm model) with a context-based learning approach for the classification of rare das including clarification requests, non-understanding signals, questions, and responses. by using suitable choices of embeddings and the inclusion of contextual history, together with a weighted cost function, we achieve good performance on these rare classes. for swda, our model achieved f1 of 0.58 and recall of 0.64 when using the immediate preceding utterance and da label, compared to f1 of 0.46, recall of 0.55 without context. we found that while gold-standard da information from context gives better performance, the performance using predicted labels can be improved by using longer contextual sequences. the resulting da tagger utilizes only minimal context of a few preceding utterances and das, rather than the whole conversation, and thus is suitable for dialogue systems in real-time, due to the left-to-right, incremental nature of dialogue. existing models which take into account the whole conversation can achieve overall higher accuracy on the general da tagging task, and so might be expected to improve our rare-class task as well, but require information about future utterances @xcite . its rare-class da outputs show good potential as features to distinguish between ad and non-ad patients in interaction, suggesting that they can be useful within tools to aid in diagnosis while providisolate the improvement gained specifically by our focus on rare class das. unfortunately this is not currently possible, as the ccc corpus has no transcripts tagged in this way. in future, we will improve the performance of our rare class da tagger with the inclusion of acoustic features from speech data. we also hope to explore more informative da sequences, including other bigram and trigram sequences, while retaining the interpretable nature of the model overall.