Where Am I and Where Should I Go? Grounding Positional and Directional Labels in a Disoriented Human Balancing Task (2022)

Abstract: In this paper, we present an approach toward grounding linguistic positional and directional labels directly to human motions in the course of a disoriented balancing task in a multi-axis rotational device. We use deep neural models to predict human subjects’ joystick motions as well as the subjects’ proficiency in the task, combined with BERT embedding vectors for positional and directional labels extracted from annotations into an embodied direction classifier. We find that combining contextualized BERT embeddings with embeddings describing human motion and proficiency can successfully predict the direction a hypothetical human participant should move to achieve better balance with accuracy that is comparable to a moderately-proficient balancing task subject, and that our combined embodied model may actually make decisions that are objectively better than decisions made by some humans.

## introduction much of the recent success in ai can be attributed to the meteoric rise of large language models (llms), such as bert @xcite and the gpt family @xcite . these language models facilitate coherent, grammatical text generation using high-dimensional representations of words, sentences, and more, that preserve similarity relations across dimensions. although pretrained on a enormous amount of text, there are many ways in which they fail to demonstrate "understanding" as commonly defined. as argued by, e.g., bender and koller (2020), these models lack knowledge of the current situational context, because that context comes from non-textual modalities. certain multimodal language models, e.g., multimodal bartlarge @xcite appear to perform better according to certain benchmarks @xcite , but there remain many important domains which for the moment appear to be out of reach for state of the art ai. consider the problem of human spatial disorientation. during extreme conditions, such as piloting a spacecraft, even expert humans are subject to gravitational transitions where they may not be able to rely on gravitational cues sensed by the vestibular system, leading to fatal accidents @xcite . even on earth, the leading cause of fatal aircraft accidents in military pilots is spatial disorientation @xcite . numerical ai models, however, with direct access to quantitative information about position and movement, can potentially determine when a human appears to be losing control and intervene, such as by telling the human what to do in order to right themselves. a successful ai partner that counteracts human disorientation to enhance task performance in real time would need to predict the intent of the human's motions, make decisions with incomplete information or under environmental uncertainty @xcite and, perhaps most importantly, foster trust in the human @xcite . these are not requirements that even the impressive benchmark performance of modern llms can meet. successful guidance of a human through language requires that the ai "embody" relations between linguistic terms and the human's situation. in this paper we combine disambiguated and contextualized linguistic embeddings @xcite from bert, with embeddings extracted from numerical ai models that are trained to predict control movements and human performance in a spaceflight-analog disoriented balancing task. unlike the bert embeddings, these latter embeddings are "situated," in that they come from models that are trained to embody a human participant's position in a phase space parameterized by angular position and velocity in the balancing task. this combined model is trained to predict the direction the human should move towards for better balance given bert embeddings that represent "thought vectors" about position relative to the balance point, and performance and motion control features extracted from the numerical models. we show that predictions made by our model "agree" on average with those made by a human with a moderate level of proficiency in the balancing task, and a deeper dive into misclassifications suggest that the model may actually be performing better in this task than the raw numerical results indicate. ## related work this paper brings together research in two distinct and to date largely disjunct areas: multimodal language grounding through human-ai collaboration, and mitigating the effects of spatial disorientation. this section discusses relevant work in these two domains and our goals in synthesizing them. the collaborative research center's situated artificial communicator project was a significant early attempt to model the integration of language and sensorimotor skills in a situated context @xcite . recent work in multimodal conversational modeling has continued similar lines of research with multimodal transformer architectures @xcite . other relatively recent work attempts to integrate neurally-encoded robotic arm control with guidance and instruction through dialogue @xcite . @xcite use unsupervised learning for concepts such as colors, names and activities by an autonomous robot. @xcite combine pcfg trees and visual feature clustering to ground video depictions of actions to linguistic labels. @xcite find that language models in a multimodal task setting learn different semantic information about objects and relations crossmodally and unimodally (text-only). importantly, though, these lines of research subsume all grounding and multimodality under combinations of language and vision, to the exclusion of other channels, and where ai and humans interact, the interaction focuses on humans guiding ai, not ai assisting humans. our work brings in modal channels directly related to human motion in a situated environment, to train an ai that ultimately assists humans to mitigate spatial disorientation. while there is a wide and varied body of research from the neuroscience and biomechanics communities on other modal information channels, such as human spatial awareness, ai has largely not been applied here. rupert @xcite presents a tactile stimulation system that provides intuitive orientation information to aircrew and operators of remote platforms and is compatible with a pilot's natural sensory system. intelligent control of such a system could help provide pilots with appropriate cues in disorienting situations, but only if human proclivites in such situations are well-understood and modeled. vimal et al. (2016) use a multi-axis rotation system (mars) device programmed with inverted pendulum dynamics to investigate learning in a dynamic balancing task about an unstable equilibrium point. subjects attempt to remain balanced by applying joystick deflections to control the motion of the device, and the authors find that subjects improve their performance by making fewer destabilizing joystick movements, and more persistent short-term joystick movements intermittently. later, they further investigate learning about different roll planes (vertical, horizontal) that disrupt the natural orientational capabilities of humans, combined with the role of gravitationallydependent otolith and somatosensory cues in the learning of the balancing task @xcite @xcite @xcite . they find that absence of gravitationally-dependent otolith and somatosensory cues degrades balancing performance. however, their findings also indicate that balance control can be enhanced in situations lacking gravitationally dependent position cues as in weightlessness, when initial training occurs with such cues present. they also observe that some participants re-learn how to balance themselves in the disorienting condition, demonstrating learning, while others do not. data from this line of research is used in this paper. recent work in this line of research has begun to use machine learning and ai techniques, providing a path forward to integrate the two aforementioned broad areas. vimal et al. (2020) group subjects performing the balancing task in the horizontal roll plane (hrp), without any gravitational cues, into performance proficiency categories using a bayesian gaussian mixture model. @xcite use the same data to train a stacked gated recurrent unit (gru) model to predict the occurrence of crashes (where crash boundaries are set @xmath0. our work extends this line of research toward modeling human behavior in the balancing task so that ai can predict and counteract disorientation. ## dataset we use data and performance proficiency labels from vimal et al. (2020) which are further explained below. additionally, we further annotate the data with grounded positional annotations and directional labels for training an embodied ai classifier that predicts optimal direction of movement. ## methodology our goal is to combine representations of motion and performance proficiency, which are learned from data directly capturing human embodiment during the mars balancing task, with linguistic representations of the position and directional concepts involved. a successful model is one which can predict the label for the best direction of motion given the current circumstances by learning correlations between motion, proficiency, and linguistic representation. the model architecture, shown in fig. 3 , can be divided into five parts: (1) data preprocessing; ## evaluation we randomly selected 12 participants from the dataset-4 participants of each proficiency. we use 38 of each participant's 40 trials for the train set and 2 for the test set. as described in sec. 4.1, we use a sliding window of 1000ms and a lookahead time of 400ms. after data processing, we end up with about 1.7 million training samples and 80,000 testing samples, for a ∼95:5 train-test split. all neural networks have 3 layers (100 units each, tanh activation), and are trained with adam optimization for 50,000 epochs. the joystick-deflection prediction model was trained with mse loss and both the performance proficiency classifier and edc were trained with cross entropy loss and a final softmax layer. to evaluate the performance/competence of the edc we examine: we also show the edc's precision, recall, and f1 for the three target labels, i.e., left, right, and center. here a "correct" answer is one where the human participant made the correct movement choice with respect to their angular position and velocity, and the model predicted the same movement choice. ## conclusion the ultimate goal of this work is to train an ai model that can give guiding cues to a human participant in real time to improve their performance in an embodied task such as the mars balancing or similar. successful guidance of a human through language requires that the ai "embody" the relation between linguistic terms and the situation inhabited by the human. here we have presented evidence that an ai model can be trained to ground directional labels to embedding-level representations of angular position and velocity, and can do so in a way that is sensitive to the proficiency level of a participant in this task, if that information is provided as input. these grounded labels can serve as cues to a human participant, as the ai considers the situation and answers "where am i?" with an answer to "where should i go?" (e.g., "i am drifting to the left. i should deflect more to the right."). our model, edc, trained on data from participants of all proficiencies, displays apparent performance on par with a somewhat proficient participant, but a deeper dive into misclassifications reveals that even though the training data itself is noisy, as the ground truth is taken to be the actual actions of the participants, even non-proficient ones, our model's apparent mislabels may actually be better decisions than those of study participants.