Predicting pragmatic discourse features in the language of adults with autism spectrum disorder (2021)

Abstract: Individuals with autism spectrum disorder (ASD) experience difficulties in social aspects of communication, but the linguistic characteristics associated with deficits in discourse and pragmatic expression are often difficult to precisely identify and quantify. We are currently collecting a corpus of transcribed natural conversations produced in an experimental setting in which participants with and without ASD complete a number of collaborative tasks with their neurotypical peers. Using this dyadic conversational data, we investigate three pragmatic features – politeness, uncertainty, and informativeness – and present a dataset of utterances annotated for each of these features on a three-point scale. We then introduce ongoing work in developing and training neural models to automatically predict these features, with the goal of identifying the same between-groups differences that are observed using manual annotations. We find the best performing model for all three features is a feed-forward neural network trained with BERT embeddings. Our models yield higher accuracy than ones used in previous approaches for deriving these features, with F1 exceeding 0.82 for all three pragmatic features.

## introduction autism spectrum disorder (asd) is a neurological disorder associated with impairments in communication that can have a life-long impact on relationships, professional success, and personal independence @xcite @xcite . although some percentage of individuals with asd are not verbal from a young age, most go on to acquire spoken language but experience challenges in social aspects of communication related to discourse and pragmatic expression @xcite . this atypicality in language has been recognized since the disorder was first named nearly eighty years ago @xcite , and unusual language usage is one of the criteria used in the primary diagnostic instruments for asd @xcite . one challenge for clinicians, however, is that there are no existing assessment tools for quantifying atypicality in discourse or pragmatics that can highlight communication deficits associated specifically with asd while ruling out those associated with unrelated language disorders. most previous work on identifying pragmatic features that index atypicality in expressive language relies on careful manual annotations of transcripts of spontaneous spoken language @xcite @xcite @xcite . deploying complex annotation schemes like these, however, is time consuming and requires training and expertise, rendering this sort of detailed linguistic analysis impractical in the clinical intervention settings in which it would be most useful. work on computational approaches for automatically identifying these features in the expressive language of individuals with asd has focused exclusively on the language of children. in addition, this prior research has generally been applied to expressive language produced in a semi-structured context with an examiner or parent rather than spontaneous conversational speech with a peer @xcite @xcite . our work addresses these aforementioned shortcomings in the previous work on pragmatic expression in asd. in this paper, we describe an annotated corpus of conversations between adults with and without asd and their neurotypical interlocutors as they engage in several collaborative tasks. using this corpus, we investigate the degree of politeness, uncertainty, and informativeness in these conversations with the goal of identifying distinc-tive pragmatic features of asd. we focus on these three features in particular because they are specific, remediable, and relevant in the collaborative discourse domain. when data collection is complete, we will release the transcribed and annotated dataset to researchers who have completed their institution's human subjects training. the dataset will be unique in that it is produced by adults, a subgroup of the asd population that is both understudied and underserved. in addition, the dataset will consist entirely of spontaneous conversations with a peer, a rarity in asd datasets. to our knowledge there is no single corpus manually annotated with all three features of politeness, uncertainty, and informativeness. moreover, our corpus is already larger than any existing spoken language (as opposed to textual) corpus available for these features. with our annotated corpus, we propose several neural models for classifying utterances according to these features, and we explore whether our automated methods of generating these pragmatic features can be used to distinguish adults with asd from their neurotypical peers as effectively as features derived via manual annotation. our models outperform prior approaches to all three classification tasks, often by very wide margins. although our predicted annotations do not capture all of the between-group differences observed using the manual annotations, we see promise in our approach. ## models after the transcripts are annotated for the pragmatic features described above, we train a number of machine learning models on the annotated data, with the goal of eventually being able to bypass the manual annotations and automate the annotation process using these predictive models. the models are given the transcribed and tokenized utterance converted to all lowercase and are tasked with predicting the categorical label for politeness, uncertainty, and informativeness based on the manual transcriptions. ## conclusions and future work from the results of our study, we can see that there exist significant and quantifiable differences in pragmatic expressions between adults with asd and their neurotypical peers. moreover these dif-ferences are not fixed or consistent across all situations, but rather they may vary depending on the open-ended nature of the task, the roles involved, and the general context of the discourse. relying on manual annotations of this sort, however, would not be practical or feasible in a clinical setting or for monitoring the efficacy of an intervention. to determine whether these annotations can be carried out automatically, we introduced several potential models trained on the annotated data. although all of our models outperformed one or more of the baselines, the bert model generally is superior for all three features. none of the models, however, were able to capture the statistically significant differences we observe in the manual annotations. there is still more work to be done in fine-tuning the model to capture between-group differences which are vital to our study of the pragmatic expression of adults with asd. in our future work, we plan to extend the current study in at least three directions. first, we would like to employ different model architectures, leveraging external labeled corpora, with more systematic comparisons to see whether the differences between asd and td groups seen in manual annotations can be fully automatically derived. second, after a long hiatus, we have recently resumed collecting data, with the goal of including 20 participants with asd and 20 with typical development. third, we aim to include annotations of other pragmatic features such as coherence and dialog acts in order to examine the differences of these features between asd and neurotypical groups more comprehensively.