A bstract M eaning R epresentation for Grounded Human-Robot Communication (2023)

Abstract: To collaborate effectively in physically situated tasks, robots must be able to ground concepts in natural language to the physical objects in the environment as well as their own capabilities. We describe the implementation and the demonstration of a system architecture that sup- ports tasking robots using natural language. In this architecture, natural language instructions are first handled by a dialogue management component, which provides feedback to the user and passes executable instructions along to an Abstract Meaning Representation (AMR) parser. The parse distills the action primitives and parameters of the instructed behavior in the form of a directed a-cyclic graph, passed on to the grounding component. We find AMR to be an efficient formalism for grounding the nodes of the graph using a Distributed Correspondence Graph. Thus, in our approach, the concepts of language are grounded to entities in the robot’s world model, which is populated by its sensors, thereby enabling grounded natural language communication. The demonstration of this system will allow users to issue navigation commands in natural language to direct a simulated ground robot (running the Robot Operating System) to various landmarks observed by the user within a simulated environment.

## introduction robots are increasingly used for their potential in disaster relief and search and rescue tasks @xcite . there is a clear benefit to this, as robots can be used to provide aid and give situational awareness of the environment to people, who can remain at a safe distance and use information gathered by the robot to knowledgeably address the situation. using robots in this way has required distribution statement a. approved for public release: distribution unlimited advances in robotics; however, robots in the current paradigm are still treated more as tools-often requiring human teleoperation, which inhibits the operator's awareness of their own immediate surroundings in potentially dangerous situations. the ability to speak to a robot as one would another human teammate would reduce the training time and cognitive burden on the operator, making the collaborative response more efficient. while there have also been relevant advances in task-oriented dialogue systems, such as siri and alexa, as well as widespread interest in systems leveraging large language models such as chatgpt, these systems are limited in their applicability to physically situated tasks because they do not address grounding natural language to the physical environment of an embodied platform. in this paper, we describe a novel system architecture that supports grounded, bi-directional human-robot dialogue. this architecture is depicted in figure 1 . in the sections to follow, we first provide a conceptual overview of the system capabilities ( §2), and then detail the components of this architecture ( §3) while highlighting the novel and primary contribution of the symbol grounding components: the abstract meaning representation (amr) @xcite parser ( §3.4), which we show to be uniquely suited to distill the action primitives and their parameters in a way that can be efficiently grounded, using our updated distributed correspondence graph (dcg) @xcite grounding component ( §3.5). we then describe the demo ( §4) and detail how distinct demo modes ( §4.1) allow users to experience performance differences when the grounding component receives input from either a syntactic constituency parser or the meaning-based, amr parser. we provide a brief comparison to related work ( §5) and conclude with directions for ongoing and future work ( §6). 2 system capabilities the implemented system in this research allows a human operator to speak to a remotely located robot in natural language, providing search and navigation instructions for the robot to execute. the current system has been successfully implemented for natural language control of a clearpath husky unmanned ground vehicle (clearpath robotics, 2023) (shown in figure 1 ), measuring about 39 inches in length and weighing about 110 pounds, which autonomously executes the natural language navigation instructions. in our implementation, the husky is equipped with a lord microstrain 3dm-gdx5-25 imu, ouster os1-64 gen 1 light detection and ranging (lidar) unit, and a teledyne flir blackfly gige camera with a kowa lmvz41 high resolution camera lens. the robot computers consist of two intel i7 equipped computers with nvidia 1650ti graphics cards installed. the robot runs on the robot operating system (ros); thus, part of our research here includes creating a ros wrapper around the amr parsing component. the same ros software stack can be used either within real-world robots or in simulation, and we have implemented and tested our architecture in both environments. because connectivity and bandwidth can be limited in disaster relief scenarios, our setup does not require internet connectivity, but it does currently require a stand-alone machine to run the natural language communication interface and dialogue management capabilities (shown in the top half of the architecture diagram in figure 1 ), whereas the rest of the system architecture components run fully onboard the robot (shown in the bottom half of the architecture diagram in figure 1 ). ## system components in the following sections, we provide an overview of each of the architecture's components. we devote the most description to the primary novel contribution of this paper: the symbol grounding, which leverages an amr parser together with a dcg grounding component. ## demo description in the demo, audience members will be invited to interact with the system at a computer workstation where the audience can see a view of the robot in the simulated environment on one side of the screen and what is essentially the robot's view of the world, or its world model, on the other side of the screen. the world model pane shows a lidarderived map of the simulated environment where detected objects in the environment are represented by white boxes with pink labels. figure 4 is a screen capture of the demo workstation. in the left pane, visitors can see that the robot is approaching a set of three barrels extending out to the right, and two cones with a small bench in between them ahead of the robot and to its left. in the right pane, visitors can see a visual representation of what the robot "sees" in this same environment using its lidar and computer vision sensors. there is a snapshot of exactly what the robot sees from it's onboard camera in the small pane in the top left corner of the right pane. the rest of the right pane populates with light grey in the areas reached by the lidar that have been classified as open space; thus, there are some darker grey unknown or unexplored areas beyond the grey barrier that encloses the demo environment. the robot recognizes the three barrels, the cones and the bench. these objects are labeled with the basic object type label as well as a unique identifier number that tracks these objects in the robot's world model. for example, the robot labels the closest barrel as "barrel-4". demo audience members will be able to direct the robot to any of the objects in the scene that the robot has identified and successfully labeled thus far. ## related work this research is at the intersection of nlp, including semantic parsing and dialogue systems, and robotics. we limit our direct comparison here to similarly interdisciplinary work; see @xcite for a full review of research in robotics and language. outside of the work on the dcg grounding approach that we directly augment for amr @xcite , field robotics has largely focused on robots that receive an initial, static tasking and then operate autonomously (e.g., @xcite ; @xcite ; @xcite ), or robots that are tele-operated (e.g., @xcite ; @xcite ; @xcite ). in contrast, there is relatively little work like ours, seeking to develop robots that are able to be tasked dynamically and interactively via natural language. there are, however, several notable exceptions. @xcite describe the development of a voice-controlled fork lift. in contrast to our own research, however, the natural language instructions are more constrained to particular hard-coded commands mentioning a more limited range of objects that are classified in their world model. @xcite develop a mobile manipulator designed for space operations that is capable of accepting spoken commands. unlike both of the previously mentioned voice-controlled robots, it is important to note that our architecture aims to support bi-directional communication between the robot and the operator, such that ambiguities that might arise in changing environments can be resolved. there is also relevant research leveraging large, pretrained language models to map or translate between unconstrained natural language and the controlled planning languages of robots. song et al. (2022) utilize gpt for deciding upon the appropriate high-level plan given natural language instructions, and then use a more traditional lowlevel planning component to execute specific motor movements to specific grounded points in the environment. the high-level and low-level models are also able to communicate, such that the high-level model can be queried for new and updated plans if conflicts arise in the low-level planning model. @xcite develop their own multi-modal "embodied" language model, called palm-e, which accepts both sensor data, such as image data, and natural language text. the model outputs text data that can be interpreted as robot policies. in general, we see potential for leveraging language models in the future both for providing some apriori, zero-shot knowledge of objects that the robot might encounter in its environment, which can be used to inform the interpretation of natural language instructions, as well as for providing a likely mapping between unconstrained natural language and the constrained set of robot behaviors. however, explainability is critical for adoption of robotic systems in high-stakes tasks such as disaster relief; thus, further research enabling transparency and explainability of systems leveraging language models is needed. neuro-symbolic approaches (e.g., dipta et al. ( 2022 )) are promising for providing greater transparency. for example, @xcite develop danli, which symbolically represents subgoals as predicates on objects in the robot's world model. there is a growing body of research leveraging amr for nlu in human-agent interaction. the present research is part of a broader ongoing research effort leveraging a two-step nlu pipeline that first parses natural language into amr, which abstracts away from some surface variation, but then in a second step converts the standard-amr into a formalism called dialogue-amr @xcite . dialogue-amr is augmented to capture features of language found to be critical for human-robot dialogue, but not included in standard-amr @xcite . specifically, the dialogue-amr adds information on the input instruction's tense and aspect, and further normalizes varying expressions for a desired behavior (e.g., turn, rotate, pivot) to a single designated roleset for a particular robot behavior (e.g., turn-01). while the present research leverages standard-amr as the input to the grounding component, we will shift to using dialogue-amr as the input parse, as we expect that the further normalization will allow us to achieve comparable results with even less training data. furthermore, dialogue-amr leverages spatial rolesets from spatial-amr @xcite , which provides detailed relations for spatial relations for expressions such as in front of, which currently does not have a detailed representation with a relational concept in standard-amr. other research to augment amr for interaction includes work to further develop multi-modal, gestural amr @xcite as well as efforts to further develop aspect and modality in amr to support nlu @xcite . finally, there is research in leveraging amr parses of image captions in order to develop scene graphs, which can help agents to summarize and process visual scenes @xcite . together, all of these threads of research demonstrate ways in which amr can serve as a unified representation for making sense of multiple modalities of information. ## conclusions and future work we are currently engaged in experimentation to evaluate the amr-based grounding. our ongoing extrinsic evaluation compares natural language interaction with the current paradigm of teleoperation. specifically, we compare the time it takes for a robot operating autonomously to complete natural language instructions, using the architecture shown in figure 1 , to the time that it takes a relatively experienced person to teleoperate the robot and complete the same instruction. this comparison is made with and without the introduction of latency, which can occur when operators teleoperate a robot from distant, remote locations. the latency, or delay, between the manual teleoperation and the robot's execution of the teleoperation can be disorienting to operators (imagine, for example, if movements of your own body were delayed for some time after your brain sending the signal to move). this disorientation can cause delays in reaching the destination, an inability to reach precise locations, or even crashes. such latency does not have a dramatic effect on natural language instructions, since although these might be delayed momentarily in getting to the robot, the robot is then navigating autonomously based upon the plan expressed in language. our early results show that while autonomous navigation is generally slower than teleoperation, with anything over one second of latency introduced, the speed of autonomous navigation becomes comparable. we are also carrying out an intrinsic evaluation where we compare our architecture, with the amr parser, against an implementation with a cyk parser in order to robustly evaluate our system against the comparable system of @xcite . we will evaluate the performance in terms of the ability of each system to successfully ground a wide variety of instructions with the same training set, and we will also compare computation time and efficiency. once our evaluations leveraging standard-amr are complete, we will then turn to comparing to the use of dialogue-amr, where we expect even greater computational efficiency since dialogue-amr abstracts even further from surface variation to normalize a variety of different expressions of different behaviors into a single amr roleset designated for a robot behavior. finally, although not the focus of this paper, we are also working to update our architecture such that the intent classification and dialogue manage-ment components work more synergistically with the grounding and planning components. therefore, the system can draw upon its knowledge of the surrounding environment to support more humanlike conversational repairs in cases of ambiguities and miscommunications. for example, if the system encounters the well-formed instruction, move to the barrel on the right, but there is no barrel grounded on the right and instead a barrel grounded on the robot's left, then that information from the grounding component can support generation, via amr, of a targeted clarification question, such as i don't see a barrel on the right; do you mean the one on the left? this requires a level of intercommunication of the components that we currently have not achieved. in this demonstration of our research, we show that amr-based grounding of natural language instructions allows our system to successfully ground and execute instructions with a range of linguistic phenomena, including light verb constructions, coreference, and spatial relations. although these phenomena are arguably complex for grounding and have proven to be challenging for the existing state-of-the-art systems, they are commonplace in natural language; thus, we simply must have systems that can handle such complexity reliably in disaster relief scenarios. in the demonstration that we offer, visitors will be able to explore this firsthand to see how our system addresses these challenges by grounding the meaning of the instructions, rather than just the words of the instructions.