Automated Scoring of Clinical Expressive Language Evaluation Tasks (2020)

Abstract: Many clinical assessment instruments used to diagnose language impairments in children include a task in which the subject must formulate a sentence to describe an image using a specific target word. Because producing sentences in this way requires the speaker to integrate syntactic and semantic knowledge in a complex manner, responses are typically evaluated on several different dimensions of appropriateness yielding a single composite score for each response. In this paper, we present a dataset consisting of non-clinically elicited responses for three related sentence formulation tasks, and we propose an approach for automatically evaluating their appropriateness. We use neural machine translation to generate correct-incorrect sentence pairs in order to create synthetic data to increase the amount and diversity of training data for our scoring model. Our scoring model uses transfer learning to facilitate automatic sentence appropriateness evaluation. We further compare custom word embeddings with pre-trained contextualized embeddings serving as features for our scoring model. We find that transfer learning improves scoring accuracy, particularly when using pretrained contextualized embeddings.

## introduction it is estimated that between 5% and 10% of the pediatric population will experience a speech or language impairment @xcite . diagnosing these impairments is complex, requiring the integration of structured assessments, medical history, and clinical observation, and there is evidence that language impairments are frequently misdiagnosed and underdiagnosed @xcite @xcite . as a result, there is a need for tools and technolo-gies that can support efficient and remote screening for language impairment. however, developing methods for automatically scoring the subtests used to diagnose language disorders can be challenging because of the very limited amount of labeled data available for these subtests from these special populations. in this paper, we focus on a task we have adapted from the formulated sentences (fs) subtest of the clinical evaluation of language fundamentals 4 (celf-4), one of the most widely used language diagnostic instruments in the united states @xcite . in the celf-4 formulated sentences task, a child is presented with a target word and an image, and must use that word in a sentence about that image. poor performance on this subtest is strongly correlated with expressive language impairments. responses are scored on a scale from 0 to 2; a sentence assigned a score of 2 must correctly use the target word, be a complete and grammatically correct sentence, and relate to the content and activities shown in the image. reliable manual scoring can be difficult and time-consuming because of the large of number of factors that must be considered. this degree of subjectivity, together with the task's important role in identifying expressive language impairments, make automatic scoring of the formulated sentences subtest particularly worthwhile. this paper makes the following contributions: ## related work scoring formulated sentences in terms of syntactic correctness can be analogous to the more common task of grammatical error detection (ged), in which points are deducted for each grammatical error detected in a sentence or text. the state-ofthe-art approaches to ged use a supervised neural sequence labeling setup to detect errors trained on artificial data @xcite . performance on this task can generally benefit from using a large size of high-quality training data, but collecting large quantities of such data is expensive. data augmentation can increase the amount and diversity of training data, provide additional information about the representations of sentences, and improve performance on the ged task. the current state-of-the-art ged trains on artificially generated data produced via error induction. one traditional way is to use the patterns learned from annotated learner corpora and apply them to grammatically correct text to generate specific type of errors, such as noun errors @xcite and article errors @xcite . more recently, artificial training data is typically generated by using machine translation, where the source text is error-free text and the target is ungrammatical text @xcite . deploying vectorized representation of word and sentence is now a ubiquitous technique in most nlp tasks. incorporating word embeddings as features can provide another possible solution in lowresource scenarios. the current state of the art ged is achieved by using bert embeddings to capture the contextual information. @xcite compare using elmo, bert and flair embeddings on several publicly available ged datasets, and propose an approach to effectively integrate such representations to detect grammatical errors. our work is inspired by this prior research on using machine translation to generate artificial data and comparing the influence of task-specific versus generic embeddings. although these methods are typically trained on second language learners' data in essay writing tasks, our goal is to seek a general representation of the syntactic and semantic representation of a single sentence in a constrained domain by children who are l1 speakers but may have deficits in expressive language. given the very limited amount of clinical data, however, we make the assumption that the types of errors language learners make can be leveraged to evaluate formulated sentences responses, an assumption that will be empirically validated in future work with our clinical dataset. ## data in this section, we describe the non-clinical formulated sentences dataset we have collected, as well as two other publicly accessible datasets, ms-coco @xcite and fce @xcite , which we use to train embeddings and generate artificial training data. formulated sentences (fs) dataset using our own stimuli designed to mimic the properties of the celf-4 formulated sentences (fs) stimuli, we collected 2160 sentences from amazon mechanical turk workers and scored the responses according to the published guidelines for the celf-4 fs task, which rely on syntactic grammaticality and semantic appropriateness given the image. each of the 24 numbered stimulus words was manually selected by a speech language pathologist in order to be comparable to the corresponding celf-4 fs stimulus word in terms of part of speech, age of acquisition, and phonological complexity. the participants were recruited on amazon mechanical turk (amt) and directed to take the test within the online survey platform qualtrics @xcite , as required by the affiliated university's institutional review board. our fs data collection effort is composed of three sub-tasks: each participant was randomly assigned to take one of the three sub-tasks. the participant was instructed to view a sample stimulus and response and was then asked to take two trial stimuli to ensure they were familiarized with the test format and environment. since there are intra-relationship between the three tasks, a participant was able to complete only one sub-task to avoid covariate effects. there were 30 participants for each task, with 24 stimuli in each test, resulting in 2160 sentences (24 stimuli * 3 tasks * 30 participants) included in the dataset. the collected sentences were scored by four native speakers of english by giving a score of 0, 1, or 2. a score of 2 indicates that the sentence fully expresses the content of the image by using the given word without any grammatical errors. if there is one grammatical error, or the sentence only represents unimportant details of the image, the sentences is marked as 1. if there are two or more grammatical errors, or if the content is unrelated to the image, it is assigned a score of 0. each grader was assigned to evaluate 2 sub-tasks, and the average pairwise kappa between the graders was 0.625. when there was a disagreement between two graders, the third grader was recruited, and the final score was the majority of the three graders. integrating a target word into an image description requires more complicated linguistic competence than using the target word to make a sentence or having a free choice of vocabulary in describing an image. therefore, task 1 is considered to be more challenging than task 2 or 3. as shown in ## results the experiments demonstrate a marginal increase in precision and substantial improvement in recall for the sentence formulation tasks. although the three tasks all involve making sentences, each task tests different aspects of linguistic knowledge. therefore it is more valuable to investigate the model performance on individual tasks, presented in on the test sets of task 1 and task 2, the best performing model is transferred sfs (task 1: @xmath0.81; task 2: @xmath1.92) with bert. for task 3, the best model is trained on the larger scoco dataset (@xmath2.82) with bert embeddings. task 2 and task 3 have marginal improvement compared with the baseline. however, task 1, which is the actual parallel to the celf-4 formulated sentence task used to diagnose language impairments in children, shows a substantial improvement in performance. the improvement by incorporating bert embedding achieves the best performance in all the three tasks individually. these experiments demonstrate that transfer learning provides a beneficial addition for evaluating formulated sentence tasks. the language representations trained on a large general dataset allow the model to acquire a better representation of linguistic knowledge. overall, we find that the model with transfer learning and bert embeddings achieves the largest improvement across all three tasks. ## discussion transfer learning is generally used in situations in which the related task has more training data than the problem of interest, and the two tasks share similarities in structure. in this paper, we train an mlp model on two artificial datasets (sfs and scoco) and improve the performance of formulated sentence scoring task. the two datasets are similar in terms of lexicon variations, syntactic structures, and semantic expressions. they are generated by using the same machine translation models. the difference between these two sets is in the number of sentences contained. since the number of correct sentences used for generated scoco is 14 times larger than that used to generated the sfs set, the final data of scoco is much larger than sfs. embedding representations can capture underlying meanings and relationships. different embeddings trained on distinct datasets may focus on particular aspects of linguistic and context information. we use two pre-trained contextual embedding and a customized embedding trained by using the coco captions, which is much more similar to the data we intend to evaluate. the results show that bert generally outperforms others in all tasks. word2vec embeddings achieve results comparable to those of using bert and outperform elmo on task 2. for the tasks involving sentence-level semantic meanings, however, its performance is inferior to the two contextualized representations. the results we have presented here point the way to new approaches for automatically scoring tasks used in clinical diagnosis of language impairments, where labeled data for training models is typically scarce. in our future work, we will apply these models to data we are currently collecting from children with language disorders, autism spectrum disorder, adhd, and typical development. although there are differences between the populations studied (mturk workers vs. children) and the modalities in which the responses were (written vs. spoken), our findings demonstrate the robustness of our methods even in the presence of domain or modality mismatch.