Measure Children’s Mindreading Ability with Machine Reading (2023)

Abstract: Recently, much research in psychology has benefited from the advances in machine learning techniques. Some recent studies showed that it is possible to build automated scoring models for children’s mindreading. These models were trained on a set of manually-labeled question-response pairs, which were collected by asking children to answer one or two questions after a short story is told or a video clip is played. However, existing models did not take the features of the stories and video clips into account when scoring, which obviously will reduce the accuracy of the scoring models. Furthermore, considering that different psychological tests may contain the same questions, this approach cannot be extended to other related psychological test datasets. In this study, we proposed a multi-modal learning framework to leverage the features extracted from the stories and videos related to the questions being asked during the children’s mindreading evaluation. Experimental results show that the scores produced by the proposed models agree well with those graded by human experts, highlighting the potential of the proposed network architecture for practical automated children’s mindreading scoring systems.

## introduction in the field of psychology, the cognitive process of inferring others' mental states through the observation of their actions and verbal expressions is commonly referred to as "mindreading". this intriguing phenomenon has garnered considerable attention from various disciplines, including psychologists, neuroscientists, economists, and computer scientists, over the course of many years @xcite . research suggests that children who exhibit exceptional mindreading abilities tend to possess a healthier psychological 1 our code is available at https://github.com/ manic-dolphin/emnlp2023-unifm-mindreading . the same question: how do you interpret the actions of the man in this clip? test b videos (or stories) questions watch the videos (or listen to the stories) test a questions test b questions answer the question response a: he is lying. response b: he is deceiving. (question, response) (question, response) bert score: 1 ❌ score: 1 ❌ (question, response a) + video (or story) (question, response b) + video (or story) score: 2 ✓ score: 0 ✓ unifm previous method our framework ⓵ ⓵ ⓶ ⓹ ⓷ ⓸ ⓺ ⓺ ⓹ ⓸ figure 1: suppose there are two different psychological tests that contain some identical questions. previous methods only considered question-answer pairs as input to the model. therefore, when faced with questionanswer pairs as shown in the figure, the model failed to learn how to score these two pairs since the same question and the children's answers were close. however, our framework can handle such situations. by incorporating the background information from psychological tests into the input, the model can utilize the additional information to learn how to distinguish semantically similar question-answer pairs, thus avoiding incorrect evaluation results. in general, our framework can be extended to multiple related datasets, requiring the training of only two models: one for cases with text-only input and another for cases with video information included. these two models are unified within the unifm framework. presently, standardized open-ended psychological tests are available to assess children's mindreading ability. these tests typically involve children responding to specific questions based on meticulously crafted vignettes prepared by experts @xcite . alternatively, assessments may involve the use of short clips extracted from comedic sources @xcite or animated content @xcite . however, evaluating children's responses to these test questions necessitates manual assessment by extensively trained experts. this process is both labor-intensive and costly. some work @xcite have attempted to tackle this challenge by integrating the aforementioned psychological tests and developing automated scoring systems to evaluate children's mind-reading ability. they constructed the mind-ca dataset @xcite , subsequently performed finetuning on a transformer-based pre-trained model using the mind-ca dataset, leading to noteworthy advancements in automated assessment of children's mind-reading abilities. despite previous research has achieved promising results, the methods employed still exhibit certain limitations. these models only considered the test's questions and children's responses as input, which brings two drawbacks. firstly, different psychological tests may share identical questions, using this method leads to unavoidable errors during evaluation of the model, which restricts the model's generalizability to other related psychological test datasets, limiting its applicability. secondly, disregarding the background information from psychological tests hampers the potential improvement in the model's evaluation performance. hence, it is crucial to acknowledge the relevance and utility of this contextual information. figure 1 illustrates an example that explains why our framework has the ability to extend to other related psychological test datasets. to address the aforementioned issues, we firstly incorporate the psychological tests' background information into our model's input. this background information includes the complete story text that is read to the children during the psychological tests, as well as the silent comedy clips that are shown to the children. in the current context, the input may involve multiple modalities, such as the combination of clips and question-answer pairs. moreover, considering the need for effective inter-action between the background information and the question-answer pairs, we secondly introduce some effective methods from machine reading comprehension. this allows the model to not only fully utilize the background information but also seamlessly unify different input types within a single framework. overall, our proposed method unifm (unified framework for measuring children's mindreading ability) can effectively addresses the issues encountered by previous models. experimental results demonstrate that our model outperforms previous approaches, and the incorporation of multimodality information significantly enhances its performance. moreover, in comparison to previous methods, unifm exhibits the capability to handle a broader spectrum of psychological testing scenarios, thereby showcasing its potential for development into an automated, cost-free, online scoring system for evaluating children's mindreading ability. ## related work this section provides an overview of the relevant literature that forms the foundation of our research. firstly, we present a concise introduction to the concept of mindreading and discuss two standardized psychological tests that were employed to construct the training dataset for automated scoring systems. secondly, we provide a brief review of studies that have explored the integration of nlp techniques with psychology, highlighting the research endeavors focused on developing automated scoring systems. throughout this paper, our primary objective is to compare our model with these existing systems. finally, we present an overview of methods employed in the machine reading comprehension task, incorporating techniques proposed in prior studies to construct our framework. ## method in this section, we provide a detailed description of our proposed method. as mentioned earlier, our framework requires the training of two distinct models to accommodate different input scenarios. hence our model is designed to handle two subtasks: the strange stories task and the silent film task. different tasks corresponds to different inputs. for the strange stories task, we utilize the source text of the "strange storie" and the corresponding question-response pairs as input to our model. on the other hand, for the silent film task, we employ the clips from the "silent films" and the related question-response pairs as input. given the inclusion of information from both the language modality and the visual modality in the input, we employ specific pre-trained models to encode this information. these pre-trained models serve as robust encoders that extract meaningful representations from the input data. subsequently, in order to facilitate the interaction between the representations of the text (or c3d or bert encoder question response value ⨁ duma layer multi-head attention features story or extracted frames … linear linear linear linear linear linear key query query key value linear linear decoder softmax score 1 score 2 probability distribution figure 2: the architecture of our model. for inputs from different psychological test backgrounds, we need to train two models with only differences in the encoder architecture. specifically, if the input consists of stories and question-answer pairs, we employ bert for encoding to obtain hidden representations. if the input includes videos and question-answer pairs, we use c3d to encode the video. once we obtain the hidden representations of the input, we utilize a dual-tower attention mechanism to interact and align these features, which is a unified process. after aligning the features, we concatenate and aggregate them, and finally decode them into a probability distribution of scores, completing the automated assessment of children's mental interpretation ability. clips) and the representations of the questionanswer pairs, we employ the "dual-tower" architecture (duma layer) that borrows from duma method @xcite . the duma mechanism enables our model to achieve effective alignment between different pieces of information, resulting in an integrated representation that captures the essential information of the passage (or the clips) and the associated question-answer pairs. this integrated representation is subsequently mapped into a probability distribution, serving as a measure of the model's confidence or certainty in assessing mindreading abilities. figure 2 provides an detailed overview of the structure of our framework. in the following subsections, we discuss in more detail our method. in section 3.1, we describe two sub-tasks, and in section 3.2, we provide a detailed description of the model architecture. ## ablation study in this section, we will dive further into the impact of the duma layer on our experimental results in the strange stroies task. additionally, we will explore the significance of incorporating visual information in improving model performance in the silent film task. we will also investigate the influence of the duma layer on aligning multimodality information. to investigate whether duma laer would benefit the model's performance, in the strange stories task, we remove the duma layer in our model, train and test it on the dataset d ss . similar to the strange stroies task, in the silent film task, we train a unifm sf model without duma layer on the dataset d sf . results in the ## statistical significance analysis to confirm that our method indeed improves the model's performance, we conduct the paired t-test on the strange story task and the silent film task. specifically, each pair consists of two models. we train and test these two models multiple times, each time with a fixed random seed shared by both models. we record the results of the models on the test set and then calculate the p-value based on the results obtained from multiple tests. we investigate the influence of the number of duma layers. we only change the number of duma layers and calculate the accuracy. the results are shown in figure 3 . in the strange stories task, our model show good performance both when @xmath0. when @xmath1, model's performance drop dramatically. it is not strange that increasing the number of duma layers does not bring promising improvement: there are only five different types of stories in the strange stories task, interacting the text of stories and questionresponse pairs once is enough to capture the hidden information. stacking too many layers makes the model harder to train and may lead to the overfitting problem. these analyses are also applicable to the silent film task. ## conclusions in this paper, we propose a new framework unifm for automated scoring children's mindreading ability. we fuse additional multimodality information from mindreading tests into the model's input. different from previous methods, this novel frame-work can be used to train on more than one psychological test dataset. experimental results have shown that our model achieves great performance on different tasks and outperforms the previous methods. in addition, the case study indicate that our models have the potential to be transferred in other scenarios.