Adversarial Text Generation using Large Language Models for Dementia Detection (2024)

Abstract: Although large language models (LLMs) excel in various text classification tasks, regular prompting strategies (e.g., few-shot prompting) do not work well with dementia detection via picture description. The challenge lies in the language marks for dementia are unclear, and LLM may struggle with relating its internal knowledge to dementia detection. In this paper, we present an accurate and interpretable classification approach by Adversarial Text Generation (ATG), a novel decoding strategy that could relate dementia detection with other tasks. We further develop a comprehensive set of instructions corresponding to various tasks and use them to guide ATG, achieving the best accuracy of 85%, >10% improvement compared to the regular prompting strategies. In addition, we introduce feature context, a human-understandable text that reveals the underlying features of LLM used for classifying dementia. From feature contexts, we found that dementia detection can be related to tasks such as assessing attention to detail, language, and clarity with specific features of the environment, character, and other picture content or language-related features. Future work includes incorporating multi-modal LLMs to interpret speech and picture information.

## introduction large language models (llms), such as gpt-4 @xcite and llama3 @xcite , have demonstrated powerful general capabilities in traditional nlp tasks like rewriting and summarization @xcite . they possess two notable advantages: first, they can easily generalize to unseen tasks and specific domains using only a few in-context samples without the need for fine-tuning @xcite . second, emergent abilities such as chain-of-thought (cot) @xcite enhance llm capability by learning to derive the final answer through intermediate steps from training or in-context examples, and offer better interpretability compared to the smaller size of language models like bert @xcite . despite its powerful capabilities, llms do not perform well in dementia detection with regular prompting strategies like few-shot or cot. dementia detection via picture description aims to infer dementia status by analyzing speech recordings or transcripts @xcite . typical accuracy of llm on dementia detection lies in the range of 55-75% in our experiments and previous works @xcite , even worse than finetuning bert-liked models with around 80% accuracy @xcite . the challenge lies in the intermediate steps of dementia detection not being well-defined, and even human experts do not have a clear understanding of what kinds of language markers could be used to detect dementia accurately. without a clear understanding, humans can not write effective demonstrations of intermediate steps, which results in llms struggling to learn how to detect dementia from training or in-context examples. as such, llms may struggle to relate their internal knowledge with dementia detection. in addition, regular prompting strategies are limited by the context window length and long context understanding capability of llms @xcite , resulting in llms are not able to fully understand and effectively learn from the training set. to bridge this gap, we propose adversarial text generation (atg) to relate dementia detection with other tasks that llm may be capable of, with the guidance of the training set. atg is a perplexity-based decoding strategy inspired by previous studies using perplexity for dementia detection @xcite @xcite . as shown in figure 1 , given a training set and an instruction, atg generates a human-understandable feature context, which could be used for perplexity-based classification. summarize the above picture description. instruction dementia transcript dementia transcript dementia transcript dementia transcript dementia transcript healthy transcript llm dataset-level input + what a busy scene! here's a summary: the scene shows a household in chaos. the cookieloving boy is sneaking cookies from the jar while sitting on a stool that's about to tip over, feature context adversarial text generation llm feature context instruction ppl calculation a pair well there's a mother standing there uh uh washing the dishes and the sink is overspilling . and uh the window's open . ## method in this section, we introduce the implementation of atg for dementia detection. we first introduce a perplexity-based classifier, which enables the use of atg for classification. it classifies a pdt transcript into dementia or healthy classes based on the perplexity of an input of a transcript, an instruction, and a feature context. a high perplexity score implies the transcript is from a dementia patient, and a low perplexity score implies it is from a healthy control. then, we introduce the two objectives for using atg for generating feature context: perplexity polarization and text coherence. the former ensures that the feature context fits the healthy transcripts and unfits the dementia transcripts, while the latter ensures the feature context is meaningful. lastly, we introduce instructionguiding strategies for atg to enhance the utility of feature context and help interpret features for dementia detection. ## data and implementation details we used three speech datasets collected via the pdt task and the cookie theft picture, which has been publicly available for dementia research. adress-2020 @xcite include 108 samples for training and 48 samples for testing. human transcripts are provided in this dataset. we present our main results using this dataset, considering it is the only dataset with standard train test split, human transcription, and balanced numbers of samples for each class, age, and gender. adresso-2021 @xcite include 166 samples for training and 71 for testing. it also has a standard train test split and balanced numbers of samples for each class, age, and gender. however, it doesn't have human transcription. we transcribe the speech samples using whisper asr (large-v3) @xcite . @xcite dataset includes 548 samples, including 243 healthy and 305 dementia transcripts. it provides human transcriptions but does not have a standard train/test split and balanced numbers of samples for each class, age, and gender, and each participant may have multiple samples. we we consider the following regular prompting strategies as baselines: 0-shot, 1-shot, 5-shot, and 0-shot-cot. for 1-shot and 5-shot, we used the first one/five samples in the training set as demonstration examples. we do not consider few-shot-cot since we cannot come up with accurate cot demonstrations. the detailed promoting templates are shown in ## discussion improving speech task design of dementia detection. our findings may contribute to the design of better speech tasks for dementia detection in the future. as shown in figure 4 , 5 and, 6, most parts of our feature contexts are related to the picture information. this indicates the most effective features for dementia detection are task-dependent. in contrast, other task-independent features may be less effective. this finding highlights the importance of task design for effective dementia detection. it also suggests that future speech task design improvement may need to prioritize the search under highly controlled settings (e.g., ask all participants to talk about the same picture, topic, etc.) to effectively elicit the difference in speech and language between dementia and healthy participants. ## related work speech analysis is a non-invasive and low-cost method for dementia classification @xcite . various speech tasks are studied by researchers such as telephone interview @xcite , linguistic features @xcite , picture descriptions @xcite , speech and writing @xcite and voice assistants @xcite . recent studies explore automatic ways @xcite to analyze spoken language to achieve fast, accurate, and ecoeconomical tools for dementia detection. there is sufficient evidence showing that machine learning has the ability to distinguish between dementia patients and healthy controls via speech performance @xcite @xcite . pre-trained and large language models, such as bert @xcite , gpt-3 (floridi and chiriatti, 2020), and llama @xcite , have achieved state-of-the-art performance on a wide range of nlp tasks. recently, researchers have used these models in dementia detection. @xcite observed that fine-tuned bert models outperform feature-based approaches on the dementia detection task. @xcite proposed a new method, gpt-d, using pre-trained gpt-2 paired with an artificially degraded version of itself to compute the ratio of the perplexities in language from dementia and healthy participants. it showed perplexity could be used for dementia detection by introducing impair- ## conclusion in this paper, we propose adversarial text generation, which relates dementia detection with existing well-defined tasks. we first introduce a perplexitybased classifier, which classifies a text sequence using perplexity, enabling the atg for classification. then, we introduce the objectives for atg, including perplexity polarization and coherence. we further incorporate a variety of instructions to guide the atg in generating effective feature context. we found high-performance instructions successfully reveal the difference between healthy and dementia transcripts, while low-performance ones fail to do so. the main features that contribute to the high performance are related to the picture contents, including the environment, characters, etc, while the language-related features may provide additional performance gain. atg could be further enhanced with multi-modal llms and could probably applied to other classification tasks with limited explicit features.