Leveraging pre-trained large language models for aphasia detection in E nglish and C hinese speakers (2024)

Abstract: We explore the utility of pre-trained Large Language Models (LLMs) in detecting the presence, subtypes, and severity of aphasia across English and Mandarin Chinese speakers. Our investigation suggests that even without fine-tuning or domain-specific training, pre-trained LLMs can offer some insights on language disorders, regardless of speakers’ first language. Our analysis also reveals noticeable differences between English and Chinese LLMs. While the English LLMs exhibit near-chance level accuracy in subtyping aphasia, the Chinese counterparts demonstrate less than satisfactory performance in distinguishing between individuals with and without aphasia. This research advocates for the importance of linguistically tailored and specified approaches in leveraging LLMs for clinical applications, especially in the context of multilingual populations.

## introduction large language models (llms) are transformative in various tasks @xcite @xcite @xcite . it remains understudied how to leverage non-english llms in a clinical context such as aphasia detection. aphasia is an acquired neurogenic language disorder, most often caused by stroke, with devastating impact on one's communication abilities. most aphasia studies with nlp perspectives focus on monolingual english speakers @xcite @xcite . fewer studies with nlp methods focus on the non-english population @xcite @xcite . to bridge the gap, we leverage pre-trained llms to detect aphasia in english and mandarin chinese speakers. given llms' widely claimed adaptability and linguistic competence @xcite , we hypothesize that integrating llms would en-hance clinical diagnosis of language disorders in aphasia. aphasia in chinese speakers has recently been studied from nlp perspectives. @xcite utilized optimal transport domain adaptation to detect aphasia in chinese and french. @xcite developed an open-source python library called blabla to automatically extract linguistic features in english, chinese and french aphasia data. @xcite focused on deep learning's application to speech assessment of chinese speakers with aphasia. @xcite used llms to derive embeddings, and fine-tuned llms for detection tasks. their findings suggest that fine-tuned models outperform acoustic features and static embeddings. as far as our knowledge goes, there is no study utilizing pre-trained llms derived surprisals to detect aphasia in chinese speakers. surprisal can be calculated by the negative likelihood of a token given previous context. conceptually, it measures the unexpectedness of a sequence in a context. surprisals' cognitive plausibility has been discussed in both psycholinguistic and clinical literature @xcite @xcite @xcite @xcite @xcite @xcite . this motivates us to implement llms derived surprisals for aphasia detection in chinese speakers. we additionally compare llms surprisals in chinese datasets with those in english, given that english is a dominant language in nlp, english speakers are the most studied population in clinical contexts, and we hope to establish an interpretation baseline on how llms surprisals behave in english aphasia speakers. ## conclusion this study leveraged pre-trained llms to detect the presence, subtypes, and severity of aphasia in english and mandarin chinese speakers. our findings suggest that without fine-tuning, taking pre-trained llms off-the-shelf can already inform us how surprisals distribute in aphasic individuals whose first language is or is not english. that said, we also found that chinese llms showed less decent performance in classifying healthy control versus aphasia, and that english llms show almost chance level accuracy in subtyping aphasia. we plan to fine-tune crosslinguistic llms using aphasia datasets to improve the models' competence in clinical tasks. our study highlights the clinical application of pre-trained llms in english and non-english aphasia individuals. there is a critical need for automatic aphasia diagnosis, since manually assessing language disturbances is labor and cost intensive, especially in low-resource non-english settings. the advent of llms has the potential to advance the field of aphasia detection. as a case study of utilizing pre-trained llms in chinese and english datasets, our investigation advocates for refining clinical nlp pipelines via incorporating llms pretrained in non-english languages.