A Spoken Dialogue System for Spatial Question Answering in a Physical Blocks World (2020)

Abstract: A physical blocks world, despite its relative simplicity, requires (in fully interactive form) a rich set of functional capabilities, ranging from vision to natural language understanding. In this work we tackle spatial question answering in a holistic way, using a vision system, speech input and output mediated by an animated avatar, a dialogue system that robustly interprets spatial queries, and a constraint solver that derives answers based on 3-D spatial modeling. The contributions of this work include a semantic parser that maps spatial questions into logical forms consistent with a general approach to meaning representation, a dialogue manager based on a schema representation, and a constraint solver for spatial questions that provides answers in agreement with human perception. These and other components are integrated into a multi-modal human-computer interaction pipeline.

## introduction despite impressive recent advances of ai in specific, narrow tasks, such as object recognition, natural language parsing and machine translation, game playing, etc., there is still a shortage of multimodal interactive systems capable of performing highlevel tasks requiring understanding and reasoning. the blocks world domain, despite its relative simplicity, motivates implementation of a diverse range of capabilities in a virtual interactive agent aware of physical blocks on a table, including visual scene analysis, spatial reasoning, planning, learning of new concepts, dialogue management and voice interaction, and more. in this work, we describe an end-to-end system that integrates several such components in order to perform a simple task of spatial question answering about block configurations. our goal is dialogue-based question answering about spatial configurations of blocks on a table, in a way that reflects people's intuitive understanding of prepositional spatial relations. the system is able to answer questions such as "which blocks are touching some red block?", "is the x block clear?", "where is the y block?", etc. (where x and y are unique block labels). distinctive features of our work: (1) it is an end-to-end system using computer vision and spoken dialogue with an on-screen virtual human; (2) it did not require a large training corpus, only a modest development corpus using naturally posed spatial questions by a few participants; (3) it derives and relies on a 3d representation of the scene; (4) it models spatial relations realistically in terms of meaningful geometric and contextual constraints. ## related work early studies featuring the blocks world include @xcite and @xcite , both of which maintained symbolic memory of blocksworld states. they demonstrated impressive planning capabilities, but their worlds were simulated, interaction was text-based, and they lacked a realistic understanding of spatial relations. modern efforts in blocks worlds include work by perera et al. @xcite , which is focused on learning spatial concepts (such as staircases, towers, etc.) based on verbally-conveyed structural constraints, e.g., "the height is at most 3", as well as explicit examples and counterexamples, given by the user. bisk et al. @xcite use deep learning to transduce verbal instructions into block displacements in a simulated environment. some deep learning based studies achieve nearperfect scores on the clevr question answering dataset @xcite . common limitation of these approaches is reliance on unrealistically simple spatial models and domainspecific language formalisms, and in relation to our work, there is no question answering functionality or episodic memory. our work is inspired by the psychologically and linguistically oriented studies @xcite @xcite . studies of human judgements of spatial relations show that no crisp, qualitative models can do justice to those judgments. the study @xcite explored computational models for prepositions using imagistic modeling, akin to the current work. another study @xcite applied imagistic approach to a story understanding task and employed blender to create 3d scenes and reason about the relative configuration and visibility of objects in the scene. ## blocks world system overview fig. 1 , 2 depict our physical blocks world (consisting of a square table with several cubical blocks, two kinect sensors and a display) and the system's software architecture foot_0 . the blocks are color-coded as green, red, or blue, and marked with corporate logos, serving as unique identifiers. the system uses audio-visual i/o: the block tracking module periodically updates the block positioning information by reading from the kinect cameras and an interactive avatar, david, is used for humanmachine communication. the block arrangement is modeled as a 3d scene in blender, which acts as system's "mental image" of the state of the world. google's cloud speech-to-text api is used for the automatic speech recognition. its output is processed to fix some common mistakes in the transcripts. the avatar is capable of vocalizing the text and displaying facial expressions, making the flow of conversation more natural than with textual i/o. the spatial component module together with the constraint solver is responsible for analyzing the block configuration with respect to the conditions implicit in the user's utterance. the eta dialogue manager is responsible for unscoped logical form (ulf) generation (see subsection below) and controlling the dialogue flow and transition between phases, such as greeting, ending the session, etc. ## evaluation we enlisted 5 volunteers, including native and nonnative english speakers. the participants were instructed to ask spatial questions of the general type supported by the system, but without restriction on wording; before their first session they were shown a short demonstration of the expected kind of interaction with the system, including question-answer exchanges. each session started with the blocks positioned in a row at the front of the table. the participants were instructed to move the blocks arbitrarily to test the robustness and consistency of the spatial models. during each session they were requested to ask 40-50 questions and mark system's answers as correct, partially correct or incorrect. they were asked to indicate separately if no answer could be given due to asr errors or when the answer (regardless of correctness) seemed to be improperly or oddly phrased. the data are presented in ## conclusion and future work we have built a spatial qa system for a physical blocks world, already able to handle a majority of questions in dialogue mode. we are not aware of any other end-to-end system with comparable abilities in qa about spatial relations. our spatial language model relies on intuitive computational models of spatial prepositions that come close to mirroring human judgments by combining geometrical information with context-specific information about the objects and the scene. this enables natural user-machine interaction. the ongoing work is targeting world history-tracking to enable answering question like "where was the toyota block initially?"