Theory of Mind in Freely-Told Children’s Narratives: A Classification Approach (2023)

Abstract: Children are the focal point for studying the link between language and Theory of Mind (ToM) competence. Language and ToM are often studied with younger children and standardized tests, but as both are social competences, data and methods with higher ecological validity are critical. We leverage a corpus of 442 freely-told stories by Dutch children aged 4-12, recorded in their everyday classroom environments, to study language and ToM with NLP-tools. We labelled stories according to the mental depth of story characters children create, as a proxy for their ToM competence ‘in action’, and built a classifier with features encoding linguistic competences identified in existing work as predictive of ToM.We obtain good and fairly robust results (F1-macro = .71), relative to the complexity of the task for humans. Our results are explainable in that we link specific linguistic features such as lexical complexity and sentential complementation, that are relatively independent of children’s ages, to higher levels of character depth. This confirms and extends earlier work, as our study includes older children and socially embedded data from a different domain. Overall, our results support the idea that language and ToM are strongly interlinked, and that in narratives the former can scaffold the latter.

## introduction one key reason language is critical to us humans is that it allows us to communicate and manipulate others' mental states @xcite . anticipating what others feel, believe, and intend, is key in navigating the social world and having meaningful interactions, and language evolved as an essential tool to achieve that (see e.g. @xcite @xcite . thus, there is a strong link between language competence on the one hand, and the competence to reason about and understand others' mental states on the other; the latter is known as theory of mind (tom) @xcite . there is a long tradition of research in child development to understand how emerging competence in language and tom interact, typically with standardized tests, carried out in lab settings with younger children, often below age 7 (for overviews see @xcite . yet, researchers in child development and cognition call for more ecologically valid data to study language and tom as social phenomena @xcite @xcite @xcite . especially for tom, researchers call to also include older subjects @xcite and methods that capture a wider variety of tom skills @xcite . we argue that children's stories are a natural choice to study language and tom competence in a social context. in narrating, children draw on various linguistic skills in producing a story, for example, structuring clauses with temporal and causal connectives @xcite . furthermore, narratives are typically rich in the feelings, beliefs and intentions of story characters, that resonate well with our own @xcite , thus inviting children to leverage their tom skills in rendering these character minds. we employ 442 freely told narratives by 442 dutch children aged 4-12 in a classification task, that we approach with features encoding the linguistic skills identified as predictive for tom performance in earlier empirical work. doing so, we evaluate and extend existing work on the links between language and tom in a natural social context and for a larger age range. we employ an adapted version of character depth (cd), originating from @xcite , as window onto children's mindreading competence. for labelling, cd indicates the mental complexity of characters, from flat characters without inner lives, to characters with basic intentionality, actions and emotions, to fully-blown characters with complex desires, beliefs, and intentions. our approach meets the 'intensional re-quirement' of any nlp-task defined by @xcite , which is having a theory on the relation between input (story) and output (cd label), next to the extensional requirement, which is simply the set of stories and labels. if the aim is to model humans' cognitive abilities with nlp-tools, then drawing on established work in other fields for meeting the intensional requirement is key. earlier work has suggested that linguistic features (e.g. vocabulary complexity) play a key role, besides age, in predicting tom in natural language data (van dijk and van duijn, 2021), but was limited in scale; here we approach language and tom in narratives at scale from a nlp-perspective. our logistic classifier performs well (f1-@xmath0.71) drawing on purely linguistic features that are relatively independent of children's age. we are able to link specific features to specific cd levels: stories employing higher cd also employ, for example, more pragmatic markers, more complex words, and more sentential complementation. our results support the idea that language and tom are intertwined, and that language can scaffold children's reasoning about the social world. this paper proceeds as follows. in section 2 we reflect on relevant work, and in section 3 we elaborate on our data and labelling. we explain feature engineering and classifier setup in section 4, present results in section 5, and contextualise results in section 6. ## background few have used nlp-tools on child language to study tom, but @xcite pioneered classifying children's tom competence on two standardized tom tests, the strange stories task @xcite and silent film task (devine and @xcite . in such tests, children are typically presented a vignette containing a social situation (verbally and/or visually) and are asked to explain why a character is behaving in a certain way (e.g. being ironic), thus inviting children to refer to characters' mental states. kovatchev and colleagues labelled ~11k answers on questions as either incorrect/partially correct/correct, depending on how appropriately children referred to characters' mental states, and obtained good performance (f1-@xmath1.91) with a distilbert transformer. indeed, accurate automatic scoring is valuable for processing standardized tom tests. it can reduce the need for resource-intensive human evaluation of answers at larger scale (for example, @xcite processed tests conducted with ~1k children), and explaining how models learn to identify correct answers can further our understanding of the relation between language and tom. @xcite however do not focus on the language children use to reason about tom, although their error analysis suggests that this is worthwhile to do. for one source of confusion identified for the transformer, is that children's answers sometimes explicate what characters would say or think. this evidences a child shifting to a different perspective @xcite , which is a precursor to tom competence @xcite . a syntactic device to achieve such shifts is sentential complementation: 'character x thinks/sees/said that it is raining', and its mastery predicts children's understanding of false beliefs @xcite @xcite . yet, since it is debated whether the role of sentential complementation holds beyond the falsebelief context @xcite , it would be interesting to see whether complementation can be linked to tom in children's natural language productions where reasoning about characters' mental states is natural, like narratives. as shown in the example above, complementation does not exclusively scaffold reasoning about mental states, but also communication and perception, which arguably provide less direct access to mental states @xcite . with modern nlp-tools, complementation in natural language can be efficiently extracted and linked to children's tom performance, as @xcite have demonstrated, and we argue that this is also worthwhile for other linguistic competences. in our view, narratives are natural devices to study language and tom. @xcite ; @xcite ; van dijk and van duijn (2021); van @xcite have found that in children's narratives, increasingly complex ways to represent characters' intentions, speech and thought can be found, which is why we look beyond standardized tests, and draw on a character depth typology established in developmental work @xcite , for labelling stories (see section 3). narrative elicitation is an established way of sampling children's language skills at lexical, syntactic, phonological and pragmatic levels @xcite @xcite , but also on examining cognitive abilities, including memorizing, planning, organizing world knowledge @xcite , and tom @xcite . the narratives central in this paper result from children's free storytelling for a live audience of peers (see section 3.1), which yields a window on children's language and tom competence that is more ecologically valid. like @xcite , we classify child language, though not test answers but a smaller set of narratives, that are linguistically speaking likely more varied. we rely on logistic regression and custom features that encode earlier findings on language competence and tom to obtain explainable performance. with shapley values we compute feature importance in the game-theoretic fashion defined by @xcite . shapley values encode the expected marginal contribution a specific feature makes to a model's prediction. if a model is a function v(x) that consists of a 'team' of n features {1, 2, ...n}, then s ⊆ n denotes all possible subsets of features, including ∅. to find the marginal contribution of feature f we must compare the model's output on a given input with f included, i.e. v(s ∪ {f }), against outputs of all models implementing all possible subsets of features without f , i.e. s ⊆ n \ {f }: shapley values are calculated for each feature and each class in multiclass classification, and are additive, i.e. they sum up to the difference between the expected value and the model prediction with all features present. ## feature engineering here we describe the engineering of features that encode language competences predictive of tom competence in children. we had 205 features in total (6 custom + 199 lemmas). since the aim is to predict cd purely from textual features, our custom features must be relatively independent from age (to not predict cd from age through language) and from one another. we computed variance inflation factors (vif) for custom features and dummy-coded age groups, with the youngest group (4-6) as reference. we adopted a threshold of 5 @xcite as indicating problematic multicollinearity; all vifs were @xmath2.54, indicating that features were relatively independent. ## analysis our analysis was implemented with scikit-learn version 1.0.1 @xcite and proceeded as follows. first, we obtained an initial random 80%-20% train-test split. we chose logistic regres- precision recall f1 actor .71 (.55) .50 (.52) .59 (.52) agent .76 (.74) .68 (.70) .72 (.72) person .76 (.79) .89 (.85) .82 (.82) average .74 (.69) .69 (.69) .71 (.69) performance metrics are given in ## discussion we employed a logistic classifier on labelling character depth (cd) for 442 freely-told stories (section 3). feature engineering was used to encode key linguistic competences identified in empirical work as predictive of tom performance (section 4). the goal was to see how these features are reflected in tom as manifested by cd. cd was predicted from linguistic features only, which were relatively independent from age (section 4). we now discuss the link between specific features and cd levels in the broader context of tom, and further reflect on language and tom competence in narratives as context-dependent phenomena. we saw that stories with flat characters (actors) are identified by the model as employing less complex words, less complementation, less pragmatic markers, and lower dependency distance. in addition, the clauses and utterances in these stories seem less explicitly linked with connectives. thus, stories without clear tom competence 'in action', are also stories in which we see less advanced language competence 'in action'. our results here mostly confirm and extend existing work on tom and language, but we saw no role for social words or lexical diversity. stories in which children do not provide insight in character minds, thus where the texts concerns mostly physical descriptions, apparently solicit less complex linguistic scaffolds. a caveat for actor stories is that our results were less robust compared to other cd levels ( in agent stories, tom competence 'in action' starts to take off with characters exhibiting implicit intentions, intentions-in-action, emotions and perceptions. in the example in person stories exhibit the highest level of tom competence in that characters show explicit (complex) intentional states, related to events, actions or other characters' mental states in the storyworld. complementation indicates person stories and thus seems to scaffold tom beyond the false belief context @xcite , likely to convey desires, beliefs, and speech, as evidenced by the lemmas indicative for this class (section 5). person stories are lexically more diverse and complex, in line with earlier work on predicting tom in narratives (van dijk and van duijn, 2021): a larger and more complex vocabulary could provide better tools to grasp and represent the social world. person stories are not distinctively associated with pragmatic markers, social words, or syntactic complexity as represented in our model. yet, regarding syntax, various connectives as positive indicators suggest that person stories have more explicitly structured clauses and utterances. thus, our result partly confirms and extends earlier research (section 4), and seems robust ( we conclude with a reflection on language and tom competence in narratives as context-sensitive, yet natural language data. some reviewers remarked that tom in narratives needs a separate accompanying measure, to make sure we are really talking about a child's tom ability when we are talking about cd. there are strong reasons to think that tom is a complex, multi-faceted ability, given the many definitions of tom that exist @xcite , and the many different standardized tests that have been designed and employed @xcite . as stated in footnote 2 (section 3.2), these tests have their own limitations; benchmarking cd with an existing standardized measure yields no simple answer to the question whether we are now talking about children's actual tom. that does not make standardized tests uninformative, but contextualizes their merit: if we agree that tom (and language) are social competences, we should also test them in social contexts, not to claim superiority over but rather complement work done in controlled settings. our classroom context has as advantage regarding tom, that children feel more motivated to do a fun task, engage with narratives as natural finding place for mental state content, have freedom to explore the (social) scenario they want, and that their language has a social goal: immersing the audience in their narratives as possible worlds. this social context may stimulate children more to challenge their language skills. to entice their audience, children may leverage their vocabulary skills to refer to rare settings, uncommon objects, unorthodox characters, and special social situations which is not possible in standardized language tests like the peabody picture vocabulary test @xcite . additionally, children may also recycle complex linguistic structures and plots from prior exposure to narratives in their own narratives, to entice their audience. thus, the influence of the social context could result in more complex language use than one would expect based on age, which makes the direct relation between age and language competence in narratives less obvious. overall, our results support the link between more complex language and tom. that said, not all tom-related content requires complex language. explicating character thought could linguistically also be represented without complement, e.g. with free direct thought ('was she angry with him?') @xcite ; moreover, the words used in this thought are not complex, nor is the syntax. this example serves to illustrate the point that in our approach, our classifier makes no assumptions at the outset about the linguistic complexity of tom-related content. ## conclusion this paper aimed to disclose the relation between language competence and theory of mind in children's freely told narratives. language competence was encoded in custom linguistic features; the mental depth of story characters was a proxy for theory of mind competence 'in action'. we linked specific linguistic contexts to lower and higher levels of theory of mind in narratives. overall, we found that stories with flat, mentally undeveloped characters (actors) are linguistically less complex, compared to stories employing characters displaying intention-in-action, emotion, and perception (agents), which in turn are linguistically less complex compared to fully-blown characters with explicit intentionality (persons). we classified character depth without drawing on children's age and obtained good performance on an initial train-test split, relative to the complexity of the task (@xmath3.71). this result was fairly robust on 100 different splits, but to a smaller extent for actor stories. overall our results support the hypothesis that in children as focal point for studying language and tom development, language and tom are intertwined and reinforce each other, using data from older children obtained in social settings.