Multimodal Modeling of Task-Mediated Confusion (2022)

Abstract: In order to build more human-like cognitive agents, systems capable of detecting various human emotions must be designed to respond appropriately. Confusion, the combination of an emotional and cognitive state, is under-explored. In this paper, we build upon prior work to develop models that detect confusion from three modalities: video (facial features), audio (prosodic features), and text (transcribed speech features). Our research improves the data collection process by allowing for continuous (as opposed to discrete) annotation of confusion levels. We also craft models based on recurrent neural networks (RNNs) given their ability to predict sequential data. In our experiments, we find that text and video modalities are the most important in predicting confusion while the explored audio features are relatively unimportant predictors of confusion in our data.

## introduction humans are adept at recognizing the emotions of others. they can identify whether another person has positive, negative, neutral, or more nuanced emotions by considering their facial expressions, voice, and words. to construct more human-like cognitive systems, it is important that, just as humans do, computational systems can infer emotions of the users that they interact with. modeling confusion is relatively under-explored and can be difficult to detect computationally. confusion can occur when someone does not know how to proceed with a task or when reconciling old beliefs with confounding information. the american psychological association's dictionary of psychology defines confusion as "a mental disturbance characterized by bewilderment, inability to think clearly or act decisively, and disorientation for time, place, and person" @xcite . potential applications of a confusion-detecting agent include taskdriven dialogue chat-bots and detecting a learner's confusion in online learning environments. we present models that leverage data across several modalities -facial expressions, speech signals with prosody, and transcribed spoken languagethat not only can be used to predictively model confusion but also to extract insights with respect to which features of which modalities are clearer indicators of confusion. in this work, we answer the following research questions: rq1 how can we improve upon prior data collection methods to obtain a more precise multimodal dataset with confusion labels? rq2 how accurate of a model can we construct that classifies the degree of confusion at different points within a task? rq3 what facial, audio, and language features serve as good predictors of confusion (or a lack thereof)? detecting confusion has mostly been explored in educational settings to discern students' confusion. as moocs (massively open online courses) have become more prevalent, researchers have focused on building models that accurately detect students' confusion. defining a learner's confusion as "an individual state of bewilderment and uncertainty as to how to move forward," atapattu et al. (2020) found that linguistic-only features were highly accurate predictors of confusion. using a dataset of nearly 30, 000 anonymous posts from stanford's mooc discussion forum, they used natural language processing resources, e.g., sentiment analysis, and a manova test to extract feature importance. while atapattu et al. (2020) focused on linguistic-only features, shi et al. (2019) analyzed facial expressions to classify learners' confusion. they used statistical learning models that leveraged a combination of histogram of oriented gradients (hog) features and local binary patterns (lbps) in tandem with a prediction system, composed of a support-vector machine (svm) and a convolutional neural network (cnn). the cnn-svm had the best performance, indicating that facial expressions can be good predictors of confusion. our research differs from these past experiments in that we aim to create a multimodal model. furthermore, we incorporate additional speech analysis to craft a more richly informed predictor of confusion. the study most closely related with our own is @xcite , which experimented with a random forest classification scheme applied over discrete time intervals extracted from two-person interactions. notably, this work considered interpretable metrics such as disfluencies (like um), questions, and pauses, although these were less correlated with confusion than the bestcorrelated facial expressions. we expand upon the study by @xcite , repeating the human subject set-up of two people collaboratively solving a task over zoom. while that study had participants label their level of confusion across a 30-second interval, our research explores continuous annotation instead of discrete spans. we expect that continuous confusion labels will enable more useful reference data for classification. ## discussion given the results of the previous section, we discuss the contributions of our work driven by our initially presented questions. specifically, we make the following contributions which we next state as answers to our original research questions: rq 1: the annotation method used by @xcite involved participants marking their confusion level for every 30-second block. we improved upon this approach by implementing the confuse-o-meter website, which allowed participants to continuously annotate their confusion levels. this method for annotation was found to provide a richer dataset in which we were able to obtain confusion labels for every time-step of data. rq 2: the rnn results showed that we were able to build a model that could relatively accurately classify confusion in the test set participants. rq 3: there were inconclusive results on which facial, audio, and language features were the best predictors of confusion because different methods yielded conflicting results. however, based on our limited results, we reason that the following features may be linked with confusion: text disfluencies, pauses, questions, au01 (inner brow raiser), au02 (outer brow raiser), au05 (upper lid raiser), au17 (lid tighten), au20 (lip stretcher), au23 (lip tighten), and au26 (jaw drop). the main limitation of our work is the size of the collected dataset -with only 20 participants, it makes sense that our models, particularly the highly nonlinear rnn system, overfit to the training samples. for any choice of two participants, it is unlikely that a model trained on 18 other participants would generalize to the test participants since confusion is a complicated emotion and not all humans display it the same way. it would take a larger dataset in order to generalize to the broader population. additionally, our models predicted the not confused states more often than the confused states. the distribution of our confusion dataset is similarly unbalanced, as seen in figure 9 . ## conclusions in this study, our goal was to build a model that was capable of accurately predicting confusion and to understand which text, audio, and video features were accurate predictors of confusion. given that the rnn has low interpretability, we utilized statistical methods to accomplish the latter half of this goal. furthermore, we improved upon previous methods of data collection to allow for continuous annotation of confusion states. this design choice provided us with a more precise multimodal dataset with rich confusion reference labels across time. to computationally model the predictive label distributions and perform confusion classification, we constructed a computational model based on recurrent neural networks (rnns), which lack interpretability but proved to be reasonably accurate even with our limited data. future work will include generalizing our rnn computational model further to better handle the different modalities (in an intermediate modality fusion scheme as in @xcite ) found within our dataset, as opposed to our current method of taking the (late-fusion) weighted consensus of three separately trained modality-processing rnns. in addition, another future next step would be to repeat our study to collect a larger dataset that better represents the general population. this may also reduce the overfitting observed in our predictive confusion models. additional research could investigate dimensionality reduction techniques and alternative forms of statistical analysis to explore measured features in our data.