Learning Multimodal Cues of Children’s Uncertainty (2023)

Abstract: Understanding uncertainty plays a critical role in achieving common ground (Clark et al., 1983). This is especially important for multimodal AI systems that collaborate with users to solve a problem or guide the user through a challenging concept. In this work, for the first time, we present a dataset annotated in collaboration with developmental and cognitive psychologists for the purpose of studying nonverbal cues of uncertainty. We then present an analysis of the data, studying different roles of uncertainty and its relationship with task difficulty and performance. Lastly, we present a multimodal machine learning model that can predict uncertainty given a real-time video clip of a participant, which we find improves upon a baseline multimodal transformer model. This work informs research on cognitive coordination between human-human and human-AI and has broad implications for gesture understanding and generation. The anonymized version of our data and code will be publicly available upon the completion of the required consent forms and data sheets.

## introduction recognizing uncertainty in interlocutors plays a crucial role in successful face-to-face communication, and it is critical to achieving common ground @xcite . to accurately identify uncertainty signals, human listeners learn to rely on facial expressions, hand gestures, prosody, or silence. ai systems that aim to collaborate and coordinate with users in a human-like manner also need to understand these signs of uncertainty. to this end, in this paper, we introduce a multimodal, annotated dataset for uncertainty detection in young children. as a multimodal communicative sign, identifying uncertainty is an important and challenging task for ai systems. especially because it varies across different ages and demographics; it is sometimes verbalized and sometimes not @xcite figure 1 : a diagram of our multimodal machine learning model. after identifying uncertainty cues in the multimodal transformer, the model passes the cues onto a final multilayer perceptron classifier to output whether the child is expressing uncertainty or not. @xcite ; it brings in different modalities, and it is subtle. although it is critical, uncertainty signal recognition is understudied in younger children. in this work, we study detecting uncertainty in the setting of a counting game for children ages 4-5. we first identify potential cues of uncertainty presented in different modalities (e.g., spontaneous verbal responses, hand gestures, facial expressions, hesitation) and specifically examine the relationship between task difficulty, task performance, and exhibited levels of uncertainty. we then use these cues to inform an ensemble model, which first identifies these cues from multimodal data and then uses them to predict uncertainty (figure 1 ). this work informs research on cognitive coordination between human-human and human-ai collaboration. with this paper, we contribute an anno-tated multimodal dataset of uncertainty in children (section 3 provides details about the dataset, annotation protocol, and analyses of the dataset); we analyze the performance of multimodal transformer models in identifying uncertainty on this dataset (sections 4 and 5); and finally we present a case study on how children express uncertainty based on their age. ## related work we cover related works of uncertainty in two sections: datasets and protocols for studying uncertainty in children, the approximate number system, and uncertainty in human-ai interactions. ## data participants a group of 68 children between the ages of 4 and 5 years old (m @xmath0;0; sd @xmath1.88 months; 28 females) was recruited through lookit, an online platform for developmental studies @xcite . thirty-six parents identified their child as white, six as asian, three as hispanic, latino, or spanish origins, and the rest as multi-racial. all but three parents reported having a college degree or higher level of education. after completing the study, compensation was sent in the form of a $5 gift card. each child participated in 30 trials which are, on average, 8 seconds long. in total, our data is composed of 16,320 seconds of video data. task participants were given an approximate number system manipulation task adapted from @xcite designed to impact children's certainty about numerical quantities. children were presented with two arrays of dots paired with two cartoon characters (figure 2 ) and asked to guess which character has more dots. both characters and their array of dots appeared for 2500 ms before disappearing. this short display duration was chosen to ensure that children did not have sufficient time to count. children were then asked to click on the side of the screen showing the greater number of dots. children were given immediate audio feedback for each trial once they chose their response. children completed 30 trials with the following number pairs: 10:9 dots (1.11 ratio), 8:7 (1.25 ratio), 14:12 (1.17 ratio), 10:8 (1.13 ratio), 9:6 (1.5 ratio), and 10:5 (2 ratio). in half of the trials, arrays with more dots had a greater, congruent cumulative area. in the other half of the trials, arrays with the greater number of dots had a smaller, incongruent cumulative area. children were randomly assigned to either the easy-first or hard-first conditions. in the easy-first condition, trials advanced from the easier trials (e.g., 10:5) to the harder trials (e.g., 10:9) in a staircase order following the design of @xcite . whereas in the hard-first condition, trials move in reverse order from hard ratios (e.g., 10:9) to easy ratios (e.g., 10:5). annotation procedure annotators first watched the video muted so as not to be influenced by the vocal feedback from the task since whether or not the child answered right or wrong may lead them to over/under-interpret certain cues. during this first watch, they marked all present physical cues as indicated by the protocol. on their second watch, annotators unmuted the video, and marked all verbal cues. if the cue was not present, the corresponding cell was left empty. ## approach with the goal of predicting uncertainty from multimodal signals, we conducted experiments with three approaches: learning uncertainty from pro-posed cues, a multimodal transformer-based model, and an ensemble learning approach that first predicts cues from the multimodal features and then predicts uncertainty from those cues. ## experimental evaluation in order to determine the viability of a multimodal uncertainty prediction model, we detail the results from each of our computational experiments and models. the implementation details of the experiments and models are given in the appendix c. baselines we employed two baseline models for comparison. the first is a simple multimodal neural network that separately processes video, au- model f1 mae r 2 no cue basic .7011 .3603 -.2419 adult .6397 .8553 -2.7119 mult .8120 .2307 -.1391 cl + w .8216 .3076 -.5412 cue ensemble .8366 .2222 -.1250 metrics in our experiment, we utilized three key metrics to evaluate the performance of our model. the weighted f1 score was employed to account for any class imbalance and provide a more comprehensive assessment of the model's precision and recall. mean absolute error (mae) was used to measure the average magnitude of the errors in our predictions, illustrating the model's ability to minimize deviations from the actual values. lastly, the r-square statistic was employed to quantify the proportion of variance in the dependent variable explained by the model, offering insight into the overall goodness of fit and the model's explanatory power. ## conventions of expressing uncertainty: a case study in different age groups from our annotated data, we find that for older children (> 2150 days old), less parental guidance is present, faster decision-making is observed, less diverse facial expressions are present, and more verbal cues are present while expressing uncertainty, which increases the performance of the models for 5-year-olds in the developmental psychology literature, nailbiting is either found to be an acquired habit or related to states of nervousness @xcite @xcite clanahan, 1995; @xcite . this behavior is hard to classify as stress-induced or uncertaintyinduced. hence, a context-dependent analysis of the person using skeletal features can help decide. pointing (see figure 7 ) is another contextdependent occurrence. when the child is uncertain, the pointing to options also becomes ambiguous, and the parent needs to ask a follow-up grounding clarification question, such as "which one?". this behavior pattern involves spatial placement of options and understanding the boundaries between them. younger children prefer ambiguous pointing gestures to conventional and visible cues of uncertainty. this type of declarative pointing is observed to be a way of engaging with the parent, pointing to a theory of mind (tom) understanding by the children @xcite . skeletal and tom modeling can help make prediction performance better. another behavior pattern is social facilitation. younger children prefer to be together with their parents while solving tasks. older children follow verbal conventions and reduce the vividness of their facial expressions, while younger children exaggerate their facial expressions and rely more on social facilitation factors. similar behavior patterns happen in adults in a competitive atmosphere where social facilitation has different effects on an individual's facial expressions @xcite . tom and multi-party dialogue modeling can increase the performance of uncertainty understanding models. uncertainty is context-dependent -some children are naturally more fidgety or shy. so predicting uncertainty on an isolated trial basis may lead to less accurate results. as a result, one interesting question is how to incorporate contextual features about the participant's personality and recent cognitive states to make more informed predictions. ## conclusion in this paper, we explored the task of predicting uncertainty in young children from an annotated dataset that we introduced with a multimodal transformer-based model. we discover that demographic and trial difficulty can affect the frequency of certain cues. moreover, trial difficulty strongly correlates with uncertainty, but trial performance interestingly does not. there is still room for improvement in task performance by transformer models, which means that more data or more complicated task setups are needed to study uncertainty properly. our dataset-which we make available for research purposes-and protocol provide future researchers with additional tools to predict uncertainty using multimodal cues to facilitate humanhuman and human-ai dialogue.