Multi-modal Intent Classification for Assistive Robots with Large-scale Naturalistic Datasets (2021)

Abstract: Recent years have brought a tremendous growth in assistive robots/prosthetics for people with partial or complete loss of upper limb control. These technologies aim to help the users with various reaching and grasping tasks in their daily lives such as picking up an object and transporting it to a desired location; and their utility critically depends on the ease and effectiveness of communication between the user and robot. One of the natural ways of communicating with assistive technologies is through verbal instructions. The meaning of natural language commands depends on the current configuration of the surrounding environment and needs to be interpreted in this multi-modal context, as accurate interpretation of the command is essential for a successful execution of the userâs intent by an assistive device. The research presented in this paper demonstrates how large-scale situated natural language datasets can support the development of robust assistive technologies. We leveraged a navigational dataset comprising >25k human-provided natural language commands covering diverse situations. We demonstrated a way to extend the dataset in a task-informed way and use it to develop multi-modal intent classifiers for pick and place tasks. Our best classifier reached >98% accuracy in a 16-way multi-modal intent classification task, suggesting high robustness and flexibility.

## introduction paralysis is a loss of motor function to varying degrees of severity often resulting in severely reduced or complete loss of upper and/or lower limb control. such impairments reduce the quality of life for millions of people affected by paralysis @xcite and increase their dependence upon others to perform day-to-day activities including self-or 0 work done while at melbourne university. the utility of the iad depends critically on the efficiency and effectiveness of the communication with the user. one of the natural ways of instructing the iad is through verbal communication. it is important to recognize that a majority of patients suffering a loss of limb control retain the ability to speak, albeit impaired in some cases. modern voice controlled iads such as wheelchairs @xcite , smart home appliances and assistive anthropomorphic robots @xcite are still limited to a pre-defined set of instructions that the user can choose from. this requires the user to explicitly dictate each individual action leading to the final goal rather than just stating the desired goal alone and off-loading the decision making to perform any required sequence of actions to accomplish the user's intent. consider the example in figure 1 , where a robotic assistant situated in a complex and dynamic environment is given a verbal instruction "pick up the book". while the need of a "pick" action is evident from the language command alone, possible additional actions (navigate to the book's location, or to turn around to face the book) depend on the agent and book's location, thus requiring an interpretation of the natural language command in the context of the surrounding environment. in this paper, we present a step towards bridging this gap by drawing on large, data resources, state of the art language understanding and intent classification methods. we develop a classifier that takes a higher-order task command contextualized in the current environment as input and derives the necessary set of sub-actions (intents) required to achieve the goal intended by the user. we present a scalable framework to develop such flexible natural language interfaces for iad that execute 'pick and place' tasks. specifically, we leverage al-fred @xcite , a large-scale naturalistic data set for developing indoor navigation systems, comprising diverse, crowd-sourced natural language commands and photo-realistic images, and adapt it to the pick-and-place task (section 3). we augment the state-of-the-art natural language intent classifier diet @xcite with a visual processing component (section 4). evaluation against simpler classifiers as well as exclusively text-based classification scenarios shows the advantage of joint processing of visual and language information, as well as the diet architecture. the use of large-scale naturalistic data allows to build solutions that generalize beyond the confines of a laboratory, are easily adaptable and have the po-tential to improve the overall quality of life for the user. this framework is part of a larger project intended to develop a multi-modal (voice and brain signal) prosthetic limb control. in short, our contributions are: our work is cross-disciplinary, covering both medical robotics and (multi-modal) machine learning and nlp for intent classification. intent classification is the task of mapping a natural language input to a set of actions that when executed help achieve the underlying goals of the user. as an essential component of conversational systems, it has attracted much attention in the natural language understanding community with methods ranging from semantic parsing @xcite to more recent deep learning @xcite and transfer learning approaches, with unsupervised pre-training @xcite . the on-line interactive nature of dialogue applications makes model efficiency a central objective. we build on the recent diet classifier (dual intent and entity transformer; @xcite ) which achieves competitive performance in intent classification, while maintaining a lightweight architecture without the need for a large pre-trained language models. diet was originally developed for language-based dialogue, and we extend the system with a vision understanding component and show that it generalizes to a multi-modal task setup. visually grounded language understanding addresses the analysis of verbal commands in the context of the visual environment. prior work ranges from schematic representations of the environment avoiding the need for image analysis (chen and mooney, 2011) over simplistic visual environments ("block worlds" @xcite ) to complex outdoor navigation @xcite ). the advance of deep learning methods for joint visual and textual processing has lead to the development of large-scale datasets which feature both naturalisitic language as well as images @xcite @xcite . we leverage a subset of the alfred dataset @xcite which is a benchmark dataset for learning a mapping from natural language instructions and egocentric (first person) vision to sequences of actions for performing household tasks. the commands in the alfred dataset are crowd-sourced from humans, and as such are diverse and resemble naturalistic language. the visual scenes are complex and photo-realistic, and the dataset contains tasks requiring the agent to execute complex sequences of multiple, context-dependent actions to manipulate objects in an environment that closely resembles the medical application scenario addressed in this paper. we note that we do not address the object recognition challenge in this work, but assume access to the object locations, and train intent classifiers to incorporate such information. interfacing medical assistive technologies traditional interfaces to assistive technologies involved manipulating joysticks @xcite , or verbal commands which are restricted to simple templates. the latter include very simple templates ("up", "down", "left"; pulikottil et al. ( 2018b )), or highly constrained training data sets based on command templates produced by five human annotators @xcite . in this paper, we leverage natural commands produced by thousands of crowd workers with the aim to produce a robust intent classifier amenable to natural speech input. ## data we leveraged and extended the alfred (action learning from realistic environments and directives) dataset of visually grounded language commands @xcite , for training and testing our intent classifier. alfred consists of more than 8,000 sets of scenes with unique environmental layout with a fixed set of associated movable and static objects. each scene is paired with an indoor navigation task, and contains three levels of information: (1) positional information of the agent and objects, (2) natural language descriptions of the high-level task and low-level instructions to achieve the goal, and (3) a sequence of discrete actions to be performed by the agent to achieve the goal. an example is shown in figure 2 . the visual task information comprises the positional (x, y, z) co-ordinates of the agent (agent information), and the positional information of static and interactable objects in the environment (scene information). the natural language annotation includes a "high-level task" describing the overall goal, as well as detailed low-level instructions ("low-level subtasks") on how to achieve the goal. low level instructions were provided by at least three human annotators through crowdsourcing. finally, each alfred task in the train and validation set is augmented with an "action plan" listing the sequence of actions (or intents) such as gotolocation or pickupobject required to achieve the goal in the context of the scene configuration (figure 2 , bottom). crowd workers were prompted by these action plans, so that a goldstandard utterance-intent alignment could be derived from the data set. ## models our intent classification model took vector representations of the language command and visual context as input and predicted the underlying intent as one of 16 classes. we briefly describe the representation schemes for scene and language input. afterwards, we present our proposed model, which extended a state-of-the-art language intent classifier to handle both visual and language input. ## experiments we present a series of experiments which assesses the impact of model complexity, multi-modal information as well as our data augmentation on final intent classification performance. this work focuses on robust multi-modal intent classification, and as such our experiments assume that the entity recognition and visual interpretation (such as object detection and location) have been solved externally. we discuss our contribution in the context of an end-to-end application section 6. ## discussion we leveraged and extended a large-scale dataset of indoor navigation tasks to develop an intent classification component for robotic arm control to perform "pick and place" tasks. our novel multimodal diet classifier exceeded 98% in classification performance in an "in vitro" evaluation setup. we now discuss limitations of our work as well as future directions. toward end-to-end task completion. the intent classifier will be embedded in a larger system in order to enable end-to-end task completion. in our evaluation, we assumed that visual scene parsing (including object recognition and location) as well as entity recognition in the language had been solved perfectly and externally. in an ongoing project, the presented system is integrated with these components, leveraging the recent improvements and corresponding tools and frameworks powered by advances in machine learning, robotics and data sets @xcite @xcite . this paper presented a highly accurate system which provides a strong foundation and promising starting point for endto-end integration as well as experiments under noisy conditions (e.g., malformed or ambiguous utterances, or speech recognition errors). diversity of tasks and inputs our study was constrained to "pick-and-place" tasks which (a) are conceptually straightforward and (b) are typically expressed in a fairly regular, formulaic manner. even though the underlying alfred data set was diverse and somewhat noisy due to its crowdsourced nature, future work will extend our scenario to more complex tasks. alfred includes a variety of tasks beyond "pick-and-place" and can directly support this line of work. our way of constructing multi-intent subtasks by concatenating low-level descriptions biased the data towards long descriptions and an underrepresentations of co-referential pronouns (e.g., "pick up the keys and put them in the bowl"). future work could leverage a mix of human data collection and natural language generation from language models to further augment the training data. the accuracy-flexibility trade-off. this work developed a highly accurate intent classifier motivated by the fact that efficient and reliable language understanding is paramount to effective humanrobot interaction. to achieve this, we limited the scenarios to a single task type as well as a simple but inflexible intent classification task: we exhaustively enumerated possible intents as 16 classes, thus preventing the model from meaningfully classifying an input that does not correspond to one of these categories. a more flexible system would predict a sequence of atomic intent labels of varying length. to this end, the task could be reframed as multi-label classification; or a sequenceto-sequence model could be developed to translate a natural language input into a sequence of intent labels. analyzing the trade-off between reliability and flexibility in the context of robust multi-modal intent classification for assistive technologies is a fruitful direction for future research. ## conclusion this paper presented a multi-modal intent classifier for "pick-and-place"-tasks which takes diverse natural language commands as input, and which will be incorporated into a natural language interface of an assistive robotic arm. our work will help to improve the naturalness of human-robot communication, which to-date often consists of mechanical (joystick) control or formulaic and templated language input. we showed how a large-scale naturalistic data set for general indoor navigation can be adapted to support training of a specific, highaccuracy intent classifier. we extended a state-ofthe-art natural language-based intent classifier to utilize both vision and language information. our evaluation showed the effectiveness of our data augmentation, and the importance of multi-modal signal for our task. we hope that our work motivates a wider, cross-disciplinary use of large-scale naturalistic data sets -which are becoming more ubiquitous in the nlp and ml communities -as a valuable resource for developing flexible intelligent assistive technologies.