Abstract-Our long-term objective is to develop robots that engage in natural language-mediated cooperative tasks with humans. To support this goal, we are developing an amodal representation and associated processes which is called a grounded situation model (GSM). We are also developing a modular architecture in which the GSM resides in a centrally located module, around which there are language, perception, and actionrelated modules. The GSM acts as a sensor-updated "structured blackboard", that serves as a workspace with contents similar to a "theatrical stage" in the robot's "mind", which might be filled in with present, past or imagined situations. Two main desiderata drive the design of the GSM: first, "parsing" situations into ontological types and relations that reflect human language semantics, and second, allowing bidirectional translation between sensory-derived data/expectations and linguistic descriptions. We present an implemented system that allows of a range of conversational and assistive behavior by a manipulator robot. The robot updates beliefs (held in the GSM) about its physical environment, the human user, and itself, based on a mixture of linguistic, visual and proprioceptive evidence. It can answer basic questions about the present or past and also perform actions through verbal interaction. Most importantly, a novel contribution of our approach is the robot's ability for seamless integration of both language-and sensor-derived information about the situation: For example, the system can acquire parts of situations either by seeing them or by "imagining" them through descriptions given by the user: "There is a red ball at the left". These situations can later be used to create mental imagery and sensory expectations, thus enabling the aforementioned bidirectionality. I. ROBOTS, LANGUAGE AND MODULARITY As robots grow in ability and complexity, natural language is likely to assume an increasingly central role in humanrobot interaction. Our current work is part of a larger effort to develop conversational interfaces for interactive robots ([3], [6] , [11], [8]). Robots that understand and use natural language may find application in entertainment, assistive, and educational domains. Such interactive robots are prime examples of systems where integration of numerous technologies in complex ways is required, and thus well designed modularity is necessary. One of the main challenges that one faces when designing such a system, is interfacing perceptual/motor with speech modules: existing natural language processing (NLP) systems cannot simply "plug and play". One historical reason behind this incompatibility is that the development of NLP and robotics have proceeded with relatively little interaction. NLP deals with the discrete, symbolic world of words and sentences whereas robotics the continuous and stochastic: one must confront the noisy, uncertain nature of physically embodied systems with sensory-motor grounded interaction. Current computational models of semantics used in NLP are variants of "dictionary definitions", essentially structured networks of word-like symbols. It is impossible to directly apply these approaches in any principled way to endow robots with linguistic skills since the underlying theories of semantics in symbol-based NLP provide no appropriate "hooks" for action and perception [9] . We posit that an important step towards bridging the language-robot divide is to develop new knowledge representations that facilitate cross-modal interoperability. Motivated by these concerns, we have developed a grounded situation model (GSM) which lies at the center of our proposed modular architecture. The GSM acts as a sensor-updated "structured blackboard", that serves as a workspace with contents similar to a "theatrical stage" in the agent's "mind", which might be filled in with present, past or imagined situations. It provides connection points to both speech-processing (through discrete verbal categories) as well as to perceptual/motor subsystems (through continuous / stochastic descriptions). When building conversational systems, one must select a subset of human language to be produced/comprehended. Although we may eventually want robots that can converse about a range of abstract topics, a natural starting point is to develop means for robots to talk about their immediate physical and social context. This parallels the development of semantics from concrete to abstract domains in children [10] . A central problem in connecting language and perception is the potential for mismatched levels of specificity. For example, the descriptive phrase "there is a cup on the table" and the visual observation of the cup will lead to consistent knowledge but with very different levels of specificity. The linguistic description does not provide information about the size, orientation, and color of the cup, and provides only bounds on its location (it's somewhere on the table). Visual perception in contrast will provide far more detail. How is a robot to translate between such varying sources of knowledge? More generally, sensory, motor, and linguistic sources differ in levels of specificity/ambiguity, yet must be aligned sensory verification, cross-modal belief propagation, and action.
Without considering learning, REF proposed a layered grounded situation model comprised of three layers that go from the continuous to a symbolic representation and allows the robot to understand the current situation, reason about its own experience and make predictions.
Grounded Situation Models for Robots: Where words and percepts meet
2006 IEEE/RSJ International Conference on Intelligent Robots and Systems
2006 IEEE/RSJ International Conference on Intelligent Robots and Systems
Computer Science