Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language (2022)

Abstract: Difficulties with social aspects of language are among the hallmarks of autism spectrum disorder (ASD). These communication differences are thought to contribute to the challenges that adults with ASD experience when seeking employment, underscoring the need for interventions that focus on improving areas of weakness in pragmatic and social language. In this paper, we describe a transformer-based framework for identifying linguistic features associated with social aspects of communication using a corpus of conversations between adults with and without ASD and neurotypical conversational partners produced while engaging in collaborative tasks. While our framework yields strong accuracy overall, performance is significantly worse for the language of participants with ASD, suggesting that they use a more diverse set of strategies for some social linguistic functions. These results, while showing promise for the development of automated language analysis tools to support targeted language interventions for ASD, also reveal weaknesses in the ability of large contextualized language models to model neuroatypical language.

## introduction autism spectrum disorder (asd) is a neurodevelopmental disorder with an estimated global prevalence of 1 in 100 worldwide @xcite . the majority of people diagnosed with asd today are verbal @xcite and have average or above average intellectual ability @xcite . nevertheless, young adults with asd are employed at significantly lower rates than their peers with other neurodevelopmental conditions, including learning disabilities and intellectual disability @xcite . difficulty with social communication is one of the diagnostic criteria for asd @xcite and is reported to be one of the strongest contributors to negative professional outcomes @xcite . for this reason, language skills are commonly targeted for intervention in individuals with asd @xcite , but it can be difficult to identify specific areas in need of remediation. most prior work on quantifying pragmatic deficits in asd has relied on manual analysis of speech transcripts @xcite @xcite , a process that is time consuming and requires expertise. in this paper, we use a spoken language corpus collected specifically for training automatic systems to explore social communication and pragmatics in asd. the corpus consists of transcribed conversations between adults with and without asd as they engage with neurotypical interlocutors in collaborative tasks designed to resemble workplace activities. we review the careful manual process of assigning pragmatic feature values and dialog act labels to each utterance. following recent prior work on these features in other contexts, we propose a bert-based framework @xcite for automatically assigning these values and labels. although our models achieve higher accuracy than previous neural and non-neural approaches to these labeling tasks on this dataset, we observe that models for some features show significantly weaker performance on utterances produced by individuals with asd. an error analysis with logistic mixedeffects regression reveals that these models fail to recognize unusual or idiosyncratic strategies for conveying certain social and pragmatic meanings. our results, while showing promise for the automated analysis of social communication in asd, point to an unsurprising but potentially problematic bias in models trained primarily on news and web data toward neurotypical language. ## background much of the prior work on extracting social communication and discourse features from conversa-tions has focused on dialogue acts, yielding both a large number of corpora and nearly as many distinct annotation schemes @xcite @xcite . conversational corpora, mostly written, have also been manually annotated at the utterance level for specific pragmatic features, including politeness @xcite ; uncertainty @xcite ; and informativeness, formality, and implicature @xcite . early work relied primarily on bag-of-words models with statistical classifiers, but recently transformer-based models have been used with success for many of these tasks @xcite @xcite @xcite . we follow this prior work in our use of bert @xcite for predicting feature labels. while many recent studies have explored automated analysis of language in asd, particularly in children @xcite @xcite , the most relevant to ours is @xcite , which introduced a corpus of conversations between adults with and without asd and neurotypical conversational partners, partially annotated for three pragmatic features. we go beyond this work in three ways. first, we complete the annotation and introduce a new feature, dialog act (see section 3.2), which has not been studied previously in asd language. second, we use bert directly rather than using bert embeddings for prediction. lastly, we give a statistical analysis of performance across diagnostic groups. ## method we remind the reader that the goal of this work is not to identify features that distinguish typical development from asd, as in prior work on applying nlp to language in autism (see section 2). instead, we aim to exploit known effective approaches to develop robust models for predicting linguistic features tied to social and pragmatic aspects of communication known to be impacted in asd in order to support targeted communication interventions. crucially, the models we develop must perform similarly for individuals with and without autism. we begin with three baseline classification models: majority class, where every sample is assigned the most frequent label; stratified, where labels are assigned randomly according to their distribution in the training data; and random, where labels are assigned uniformly randomly from the set of possible labels. previous work @xcite on a subset of this dataset using a different cross-validation strategy has shown that these baselines yield competitive performance to bag-of-words models and existing statistical models @xcite trained on separate corpora of written texts for these features. we compare these baselines to neural models trained with bert @xcite using machamp (van der goot et al., 2021) and its default parameters for classification tasks. 1 we did not explore statistical models here since neural models were shown to be substantially better in @xcite . to learn how models might perform differently for participant groups whose linguistic features are potentially atypical, we measure model performance separately for each speaker group: asd eps, td eps, cps when interacting with asd eps, and cps when interacting with td eps. model performance was indexed with raw accuracy and weighted f1 scores. prior work has demonstrated that incorporating contextual information (in this case, previous utterances) is useful for predicting dialog act labels for both human-human @xcite and humanchatbot @xcite interactions. to see whether a similar approach will be effective in our setting, we applied an evaluation scheme of heldout transcript 2 , where we iteratively held out the data from one full transcript (i.e., the full conversation between one ep and one cp) as the test set and used the data from the remaining transcripts as the training set. this allowed us to incorporate contextual information by embedding the preceding utterance for feature prediction without enabling the models to learn individual speaker characteristics. ## results to investigate whether adding contextual information was helpful in feature prediction, we compared for each feature the performance of the models with context to those without using logistic mixed-effects regression. the dependent variable was whether the feature value predicted by the model matched the manually assigned value; the fixed effect was the model (with or without the context of the previous utterance); and participant identity was included as a random intercept to control for repeated measurements of the same speaker. though the results revealed no significant difference for the three ordinal features, including context appears to help improve model performance for dialog act (β = 0.14, @xmath0.001). all further results presented will pertain to the models trained with prior contextual information. we now turn to the question of whether there are differences in neural model performance on the utterances of asd vs. td experimental participants (eps), as well as the utterances of conversational partners (cps) of eps with asd vs. cps of eps with td. again, we applied logistic mixed-effects regression. the regression structure was similar to that described above, except that the group to which the speaker of the utterance belongs was used as the fixed effect (asd ep vs. td ep; or cp of asd vs. cp of td). without using speaker identity as a random intercept, we found a significant effect of speaker group for eps (β = -0.31, @xmath1.423, @xmath2.001) for politeness, which indicates that model predictions for politeness are more accurate for the td group; we observed no such effect for cp groups. for dialog acts, there was also significant difference in accuracy between groups for eps (β = -0.38, @xmath3.46, @xmath4.001), indicating that the models were more accurate for td than for asd utterances; a similar but weaker pattern was observed for cps (β = -0.22, @xmath5.93, @xmath6.01). no significant differences were observed for uncertainty or informativeness. running the same analysis using speaker identity as a random intercept, the significant differences for politeness and dialog acts are weaker but maintained for eps (politeness: β = -0.32, @xmath7.00, @xmath8.05; dialog acts: β = -0.32, @xmath9.15, @xmath10.05), suggesting that the observed differences in model accuracy may be driven by certain individuals, a finding that aligns with prior observations of heterogeneity in asd language. to qualitatively understand why our models might be more accurate for td utterances, we inspected some of the incorrect predictions for asd utterances. at times we observe similar unusual communication features in multiple asd subjects, while other strategies appear to be idiosyncratic. ## conclusions using a corpus of collaborative conversations between adults with and without asd and their neurotypical conversational partners, we outline a framework for automatic identification of linguistic features associated with social communication. although transformer-based models were able to achieve strong performance overall, when comparing results between diagnostic groups, we found that models fall short on the language of participants with asd, especially in cases of politeness rating and dialog act labeling. this suggests that as powerful as transformer models are in capturing certain linguistic aspects of (written) data produced by (presumably) neurotypical speakers (see @xcite for a review), they do not suffice in characterizing language in asd, a finding that has broad implications for work applying these models to any potentially atypical language.