“What is on your mind?” Automated Scoring of Mindreading in Childhood and Early Adolescence (2020)

Abstract: In this paper we present the first work on the automated scoring of mindreading ability in middle childhood and early adolescence. We create MIND-CA, a new corpus of 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14. We perform machine learning experiments and carry out extensive quantitative and qualitative evaluation. We obtain promising results, demonstrating the applicability of state-of-the-art NLP solutions to a new domain and task.

## introduction successfully navigating day-to-day social interactions often hinges on being able to recognize that others can hold different beliefs, thoughts, ideas, intentions and desires. popular culture and literature are replete with instances that require us to enter into the mind of someone else in order to explain their actions. for example, the final scene of shakespeare's romeo and juliet is made all the more poignant because the audience know that juliet is still alive and that romeo has killed himself because he had a mistaken belief that she was dead. the audience is able to understand both the state of the world (i.e.: juliet is still alive) and the mistaken belief of romeo (i.e.: juliet is dead). the ability to attribute mental states to others in order to explain or predict behavior, called "theory of mind" or "mindreading" has intrigued researchers in disciplines as diverse as psychology, philosophy, neuroscience, economics, and literary theory for four decades @xcite . the long-standing interest in mindreading is not surprising as children's performance on standard tests of mindreading has real-world significance. longitudinal studies demonstrate that children who excel at tests of mindreading are more likely than their peers to be identified as popular by classmates @xcite , have reciprocated friendships @xcite , and be viewed by their teachers as socially competent @xcite . furthermore, case-control studies show that deficits in mindreading are more common among individuals with a range of mental health problems and neuro-developmental conditions associated with poorer social functioning @xcite . given the prognostic importance of mindreading in children, in this paper we present the joint work of an inter-disciplinary team of psychologists, computer scientists, and linguists on the automatic scoring of mindreading ability in middle childhood and early adolescence. the great majority of studies on mindreading have focused on identifying the age at which preschool children recognize that others have desires, beliefs and knowledge that differ from their own @xcite . however, the past decade has witnessed increased interest in the continued growth of children's understanding of mind in middle childhood and early adolescence @xcite @xcite . there is currently no consensus on how best to measure mindreading in middle childhood and adolescence but standardized open-ended tests in which children are asked to explain the behavior of a character depicted in short vignettes @xcite or in animations or film clips @xcite are increasingly used. while promising, open-ended responses to test questions are currently manually rated by trained experts limiting the potential utility of these measures in large-scale research. this challenge is not unique to research on mindreading as open-ended text response assessments are widely-used in clinical and educational research @xcite @xcite . automated scoring systems have the potential to enhance reliability and minimize the amount of labor required to administer tests at scale. to address this problem, we gathered, digitalized, and manually rated responses to two tests of mindreading: the strange stories task @xcite and the silent film task @xcite . the new corpus, that we call mind-ca, contains 11,311 open ended responses in english from children aged 7 to 14. we used mind-ca to train and evaluate several different machine learning systems. we performed an extensive quantitative and qualitative evaluation and error analysis to measure the applicability of the automated systems. our findings demonstrate that nlp can be used to aid the work of psychologists, educational researchers, clinicians and psycho-linguists. state-of-the-art text classification systems report very good quantitative results when scoring children's mindreading capabilities. at the same time, the in-depth analysis of the performance indicates that there is still room for improvement and future work. the rest of this article is organized as follows. section 2 presents related work. section 3 provides details about the creation of the dataset. section 4 describes the experimental setup and the evaluated systems. section 5 presents the results of the automatic scoring and the qualitative error analysis. section 6 discusses the implications of the findings of this work. finally, section 7 presents the conclusions and outlines directions for future research. ## related work the extension of research on mindreading into middle childhood and adolescence has stimulated the creation of new tests because the standard tasks used to measure mindreading in early childhood (e.g., the false belief task) exhibit marked ceiling effects by age 6 @xcite . advanced tests of mindreading used in middle childhood and adolescence employ a range of stimuli (e.g., vignettes, animations, film clips) and appear to target distinct skills (e.g., perspective taking, intention attribution, explaining behavior) @xcite @xcite . despite these in-roads, research with school-aged children has revealed limited convergence between tasks (i.e., suggested tasks may not capture the same underlying ability) (warnell and redcay, 2019) and limited discriminating power in some tasks (i.e., suggested tasks may not capture sufficient variance in middle childhood) @xcite . in contrast with these results, studies of children aged between 7 and 13 years have shown consistent associations between children's performance on two tests: the strange stories @xcite and the silent film task @xcite . the strange stories task was developed for studying mindreading in adolescents and adults with autism. in this task participants answer questions about characters in short vignettes depicting instances of mistaken beliefs, misunderstanding, double-bluff or deception. supporting its validity, adolescents and adults with autism (a condition associated with impaired social and communicative ability) were less likely than 'neurotypical' controls to explain characters' behaviours with reference to their thoughts, feelings and desires @xcite . building on this work, @xcite devised the silent film task in an effort to create a measure of individual differences in mindreading suitable for use in middle childhood and adolescence. like the strange stories task, the silent film task requires participants to answer open-ended questions about a character's behavior with reference to the character's mental states but uses brief clips taken from a classic silent comedy. the two tests show convergent validity in middle childhood and early adolescence and marked individual differences in performance in each age group studied @xcite . the two tasks exhibit strong one-month test-retest reliability @xcite and the validity of these tasks is bolstered by evidence of correlations between test performance and measures of children's social competence in middle childhood and early adolescence @xcite . the strange stories and silent film tasks are promising measures with robust psychometric properties. however, like other open-ended measures in psychology, children's responses to the test questions on the strange stories and silent film tasks are manually rated by trained researchers limiting the potential utility of these measures in large-scale research for two reasons. first, manual coding of open-ended text responses (even short responses) is labor-intensive and time-consuming. second, even trained researchers may change the way they apply scoring rubrics over time or make errors in applying a scoring rubric. such "drift" is not detected in standard inter-rater agreement assessments and potentially undermines the reliability of manual ratings @xcite . the creation of an automated scoring platform for the strange stories and silent film tasks raises the possibility of rapid, reliable assessment of individual differences in mindreading ability across middle childhood and adolescence. automated scoring is not a novel application within nlp. it is usually deployed in the educational domain. @xcite , @xcite present different work on automated scoring. the asap foot_0 and asap2 foot_1 as well as the semeval 2013 shared task 7 @xcite helped popularize the topic within nlp researchers. while the majority of the work is focused on essay grading, @xcite implement and integrate baselines of nlp for the automated coding of short text responses. @xcite describe an automated scoring and feedback system that can help students revise and improve their scientific argumentation. madnani and cahill (2018) present a position paper about using nlp for automated scoring. they argue that automatic scoring is a complex problem with multiple perspectives, and "stakeholder" implications. as such, they suggest that designing a successful automated scoring solution requires collaboration between nlp researchers, subject matter experts, and score users. they also point out that the nature and importance of automated scoring requires a much more thorough evaluation and analysis. in this paper we present the first work on the automated scoring of children's open-ended responses to tests of mindreading ability. we follow the best practices in automated scoring in order to ensure the applicability and quality of the proposed solution. ## dataset creation in this section, we present the process of creating a novel nlp resource -the mind-ca corpus. section 3.1 describes in more details the two mindreading tests that we use: the strange stories task and the silent film task. section 3.2 details the corpus creation process, the format of the final corpus, as well as some statistical information. ## experimental setup the main objective of this work was to create and evaluate an automated solution for scoring children's mindreading ability from open-ended responses to two standardized tests. a successful solution will facilitate large scale psychological research on mindreading. we formally defined the task of automatic scoring as a classification problem. we chose classification over regression taking in account the specifics of the original tasks and the data, and the need to have item-level scores for future research. for our experimental setup, we selected four different machine learning architectures. the systems were chosen based on two main criteria: 1) they represent qualitatively different approaches to the formalization and processing of the data; 2) they are sufficiently lightweight in terms of computational requirements to be implemented in a web-based end-to-end application for researchers in psychology. the systems are: the systems were implemented using scikit-learn @xcite , tensorflow 2.0 @xcite , huggingface @xcite , and ktrain (maiya, 2020) libraries for python foot_4 . all models were trained on a single gpu. for each of the systems we tested both formats of the input described in section 3.2. in the response-only setup we only provided the systems with the open-ended response and the corresponding score. in the full setup the "response" column also included the question. for the bi-lstm and the self-attention based models, we tried two different ways of initialization: random initialization and initialization with pre-trained fasttext embeddings @xcite . fasttext performed much better in the first few training epochs, however after the 15th epoch the performance of the two initializations is almost identical. we report the results obtained from the random initialization. ## experimental results we performed a 10-fold cross validation on the data. we further split each training set to 75% training and 25% validation. we report the average performance across the 10 runs for each system. we evaluated the performance of the automated systems in four different ways. in section 5.1 we report the overall performance of the systems, measured in accuracy and macro-f1. in section 5.2 we discuss the performance of the systems on scoring each of the 11 different questions. in section 5.3 we analyze the variance of the performance of the systems in terms of the age of the participants. finally, in section 5.4 we present a human analysis of the incorrect predictions made by two of the systems. ## discussion of the results in section 2 we pointed out two main limitations of the manual scoring of tests for mindreading: 1) it is labour-intensive and 2) even trained researchers may change the way they apply scoring rubrics over time. the goal of our work was to to address both of these problems. to the best of our knowledge, this is the first work that applies nlp techniques to the automatic scoring of children's open-ended responses to experimental tests of mindreading. we demonstrated that stateof-the-art nlp systems can obtain high accuracy on the task. we also proposed a methodology for an in-depth evaluation and error analysis. we showed that while the overall accuracy and macro-f1 are very high, the performance varies significantly per question. the automated systems also faced problems when the answers were more creative and unique. the performance of the systems also dropped due to spelling and grammar errors. the in-depth evaluation and error analysis gives clear directions for improvement before an automated solution can be implemented in practice. this work, and in particular the expert error analysis also helped to re-evaluate and improve the original tasks. during the qualitative analysis trained test raters developed suggestions on how to improve the existing guidelines for human annotators. this will improve the consistency of the human annotation in further studies 7 . this paper emphasizes the advantages of inter-disciplinary work and shows the importance of incorporating in-domain experts (in our case developmental psychologists) when designing and implementing a novel nlp solution. the collaboration between researchers from different fields provides different perspectives on the problem. it benefits the formalization of the task and the evaluation of the systems. ## conclusions and future work in this paper we presented the first nlp approach for the automated scoring of mindreading in middle childhood and early adolescence. an inter-disciplinary team of psychologists, computer scientists, and linguists created mind-ca -a corpus of 11,311 responses from 1,066 children aged from 7 to 14. we trained and evaluated four different machine learning systems on the data and obtained promising results. we presented a novel methodology for evaluation and error analysis inspired by the interdisciplinary collaboration between psychologists and nlp researchers. in future work we want to move towards designing a web application that can be used to gather the responses from the children and score them automatically in an end-to-end manner. to ensure the quality of an end-to-end solution and the reliability of the automated scoring, we will further test the capabilities of the automated scoring systems and ensure that they can generalize well to novel data. we are in the process of digitalizing and scoring the responses of 1,000 more 8-to 13-year-old children, gathered in a different time span and from different schools than mind-ca. this way we will double the existing corpus size and will also be able to test the performance of the systems on further, unseen data. we are also exploring the possibilities to use data augmentation on the current corpus. data augmentation can be used to address the problem of imbalanced distribution of labels for some of the questions. in the experimental setup we plan to incorporate more extra-linguistic data into the evaluation, for example the gender of the participants and their responses to standardized language ability tests and psychopathology. finally, the analysis of the performance of the systems showed that spelling and grammar mistakes were a systematic source of error and the out-of-the-box correction systems were not well adapted to deal with children's open-ended text responses. we will examine the possibility of developing a spelling correction system designed to work with participants in middle childhood and early adolescence.