Learning Co-Speech Gesture for Multimodal Aphasia Type Detection (2023)

Abstract: Aphasia, a language disorder resulting from brain damage, requires accurate identification of specific aphasia types, such as Broca’s and Wernicke’s aphasia, for effective treatment. However, little attention has been paid to developing methods to detect different types of aphasia. Recognizing the importance of analyzing co-speech gestures for distinguish aphasia types, we propose a multimodal graph neural network for aphasia type detection using speech and corresponding gesture patterns. By learning the correlation between the speech and gesture modalities for each aphasia type, our model can generate textual representations sensitive to gesture information, leading to accurate aphasia type detection. Extensive experiments demonstrate the superiority of our approach over existing methods, achieving state-of-the-art results (F1 84.2%). We also show that gesture features outperform acoustic features, highlighting the significance of gesture expression in detecting aphasia types. We provide the codes for reproducibility purposes.

## introduction aphasia is a language disorder caused by brain structure damage affecting speech functions, commonly triggered by stroke and other factors such as brain injury, dementia, and mental disorder @xcite . depending on prominent symptoms and severity, aphasia can be classified into the eight types @xcite such as @xcite 5 in appendix. people with aphasia (pwa) can face various communication challenges due to limited language processing and comprehension capabilities, which can result in difficulties with social interaction @xcite since broca's aphasia non-fluent & comprehension v a t … … i no, [*] the traditional assessments for pwa are known to be time-consuming and costly @xcite , identifying the pathological language impairment from speech data has received great attention @xcite . clinically, a detailed diagnosis of aphasia type is imperative for proper treatment procedures; however, little attention has been paid to developing a classification model for the types of aphasia. for diagnosing various types of aphasia, analyzing the co-speech gestures of pwa can be essential. pwa often rely on non-verbal communication techniques, especially gestures, as an additional communication tool due to difficulties in word retrieval and language errors @xcite . therefore, the same word can be interpreted differently depending on the accompanying gestures for different aphasia types. for example, as shown in figure 1 , individuals with broca's aphasia show a higher frequency of iconic gestures because broca's area damage results in preserving semantic content but being impaired fluency @xcite . on the other hand, individuals with wernicke's aphasia tend to derive limited assistance from gestures since fluent but incoherent speech pathology @xcite . hence, since the meaning of the word 'i' from pwa with broca's aphasia may have more latent information, utilizing both speech (i.e., linguistic and acoustic) and gesture (i.e., visual) information is crucial in classifying aphasia types. in combining two different modalities, speech and gesture, applying the existing multimodal fusion models is challenging due to the following two limitations. first, the existing models were usually built upon emotion recognition or sentiment analysis benchmarks @xcite mainly consisting of facial information; yet, pwa struggles in using faces caused by the impaired emotion processing @xcite . second, these models tended to capture a single link across the modalities, e.g., linking happinessrelated words to smiling faces @xcite , but there can be multiple links across the modalities depending on aphasia types. (e.g., 'i' may be connected to multiple gestures based on aphasia types, as shown in figure 1 ). to address these challenges, we propose to apply a graph neural network to generate multimodal features for each aphasia type, enabling the acquisition of rich semantics across modalities @xcite . due to the heterogeneity among multimodalities in clinical datasets @xcite , a unique graph structure can help distinguish diverse patterns @xcite by learning the correlation between speech and the corresponding gestures of pwa with different aphasia. specifically, we construct a heterogeneous network that represents the relationships between speech (particularly including disfluency-related keywords and acoustic information) and gesture modalities based on their co-occurrence by using data from aphasiabank @xcite , a shared database for aphasia research. then, speech-gesture graph encoder ( §4.2) extracts three node embeddings for disfluency-related keywords, audio, and gesture tokens by aggregating information from different sources. before fusing the modalities, the gestureaware word embedding layer ( §4.3) generates textual representations sensitive to gesture informa-tion by adjusting the weights of pre-trained word embeddings with the refined representations of disfluency tokens. consequently, a multimodal fusion encoder ( §4.4) is applied to incorporate aphasiatype-specific multimodal representations to predict the final aphasia type ( §4.5). the extensive experiments show that our model performs better than the prior methods in detecting aphasia types. we find that applying the gnn can effectively generate multimodal features by capturing relations between speech and gesture information, enhancing performance. our analysis reveals that gesture features play more critical roles in predicting aphasia types than acoustic features, implying that gesture expression is a unique attribute of pwa with different types. we exemplify that the qualitative analysis based on the proposed model can help clinicians understand aphasia patients more comprehensively, helping to provide timely interventions. to the best of our knowledge, this is the first attempt that uses gesture and speech information for automatic aphasia types detection. ## related work aphasia analysis and detection. to identify the pathological symptoms shown in a narrative speech of pwa, researchers have focused on linguistic features (e.g., word frequency, part-of-speech @xcite , word embeddings @xcite ), and acoustic features (e.g., filler words, pauses, the number of phones per word @xcite ). with the advances in automated speech recognition (asr) that can make the transcription of aphasic speech into text @xcite , there have been end-toend approaches that do not require explicit feature extraction in assessing patients with aphasia @xcite . while these works reveal valuable insight into detecting aphasia, little attention has been paid to identifying the types of aphasia, which can be crucial for proper treatment procedures. considering the importance of understanding gesture patterns for identifying the types of aphasia @xcite , we propose a multimodal aphasia type prediction model using both speech and gesture information. multimodal learning for detecting language impairment. multimodal language analysis helps to understand human communication by integrating information from multiple modalities for tasks such as sentiment analysis @xcite and emotion recognition @xcite . however, previous studies simply concatenated features to identify language impairment in speech data @xcite , which lacks comprehension of the complex connections between modalities @xcite . to capture interconnections between modalities, recent studies used transformerbased multimodal models @xcite ) that can extract contextual information across modalities, utilizing factorized coattention @xcite , self-attention (hazarika et al., 2020), or cross-modal attention @xcite . while the prior work tended to capture a single link between modalities, e.g., linking the 'happy' (linguistic modality) to a smiling face (visual modality) @xcite , little attention has been paid to learning multiple links across the modalities, which can be crucial in aphasia type detection. as shown in figure 1 , speech and gesture modalities may have multiple links across the different aphasia types. hence, to distinguish aphasia types, we propose to apply a graph neural network to generate multimodal features for each aphasia by learning the correlation between speech and gesture patterns of pwa. ## aphasia dataset data collection we sourced our dataset from the aphasiabank @xcite , a shared database used by clinicians for aphasia research; the corpus information is summarized in each data is categorized into one of four types: control, fluent, non-comprehension, non-fluent, based on the ability to auditory comprehension and fluency skills, as shown in ## aphasia type detection figure 2 illustrates the overall architecture of the proposed model for detecting aphasia types. the model comprises four main components: speech-gesture graph encoder ( §4.2), gesture-aware word embedding layer ( §4.3), multimodal fusion encoder ( §4.4), and aphasia type prediction decoder ( §4.5). ## conclusion in this paper, our approach utilizing gnns proves effective in generating multimodal features specific to each aphasia type for predicting aphasia types. the extensive experiments demonstrated state-ofthe-art results, with visual features outperforming acoustic features in predicting aphasia types. we plan to provide the codes for reproducibility, facilitating further research in this area. the qualitative analysis offered valuable insights that can enhance clinicians' understanding of aphasia patients, leading to more comprehensive assessments and timely interventions. overall, our research contributes to the field of aphasia diagnosis, highlighting the importance of using multimodal information and graph neural networks. we believe our findings will drive advancements in speech and language therapy practices for individuals with aphasia.