Learning to Read Maps: Understanding Natural Language Instructions from Unseen Maps (2021)

Abstract: Robust situated dialog requires the ability to process instructions based on spatial information, which may or may not be available. We propose a model, based on LXMERT, that can extract spatial information from text instructions and attend to landmarks on OpenStreetMap (OSM) referred to in a natural language instruction. Whilst, OSM is a valuable resource, as with any open-sourced data, there is noise and variation in the names referred to on the map, as well as, variation in natural language instructions, hence the need for data-driven methods over rule-based systems. This paper demonstrates that the gold GPS location can be accurately predicted from the natural language instruction and metadata with 72% accuracy for previously seen maps and 64% for unseen maps.

## introduction spoken dialog systems are moving into real world situated dialog, such as assisting with emergency response and remote robot instruction that require knowledge of maps or building schemas. effective communication of such an intelligent agent about events happening with respect to a map requires learning to associate natural language with the world representation found within the map. this symbol grounding problem @xcite has been largely studied in the context of mapping language to objects in a situated simple @xcite or 3d photorealistic environments @xcite , static images @xcite , and to a lesser extent on synthetic @xcite and real geographic maps @xcite @xcite . the tasks usually relate to navigation @xcite or action execution @xcite and as-figure 1 : user instruction and the corresponding image, displaying 4 robots and landmarks. the users were not restricted or prompted to use specific landmarks on the map. the circle around the target landmark was added for clarity for this paper; users were not given any such visual hints. the task we address here is the prediction of the gps of this goal destination by reference to a map, which is of critical importance in applications such as emergency response where specialized personnel or robots need to operate on an exact location (see fig. 1 for an example). specifically, the goal we are trying to predict is in terms of: a) the gps coordinates (latitude/longitude) of a referenced landmark; b) a compass direction (bearing) from this referenced landmark; and c) the distance in meters from the referenced landmark. this is done by taking as input into a model: i) the knowledge base of the symbolic representation of the world such as landmark names and regions of interest (metadata); ii) the graphic depiction of a map (visual modality); and iii) a worded instruction. our approach to the destination prediction task is two-fold. the first stage is a data collection for the "robot open street map instructions" (rosmi) @xcite corpus based on openstreetmap @xcite , in which we gather and align nl instructions to their corresponding target destination. we collected 560 nl instruction pairs on 7 maps of different variety and landmarks, in the domain of emergency response using amazon mechanical turk. the subjects are given a scene in the form of a map and are tasked to write an instruction to command a conversational assistant to direct robots and autonomous systems to either inspect an area or extinguish a fire. the setup was intentionally emulating a typical 'command and control' interface found in emergency response hubs, in order to promote instructions that accurately describe the final destination, with regards to its surrounding map entities. whilst osm and other crowdsourced resources are hugely valuable, there is an element of noise associated with the metadata collected in terms of the names of the objects on the map, which can vary for the same type of object (e.g. newsagent/kiosk, confectionary/chocolate store etc.), whereas the symbols on the map are from a standard set, which one hypothesizes a vision-based trained model could pick-up on. to this end, we developed a model that leverages both vision and metadata to process the nl instructions. specifically, our mapert (map encoder representations from transformers) is a transformerbased model based on lxmert. it comprises of up to three single-modality encoders for each input (i.e., vision, metadata and language), an early fusion of modalities components and a crossmodality encoder, which fuses the map representation (metadata and/or vision) with the word embeddings of the instruction in both directions, in order to predict the three outputs, i.e., reference landmark location on the map, bearing and distance. our contributions are thus three-fold: ## related work situated dialog encompasses various aspects of interaction. these include: situated natural language processing @xcite ; situated reference resolution @xcite ; language grounding @xcite ; visual question answer/visual dialog @xcite ; dialog agents for learning visually grounded word meanings and learning from demonstration @xcite ; and natural language generation (nlg), e.g. of situated instructions and referring expressions @xcite . here, work on instruction processing for destination mapping and navigation are discussed, as well as language grounding and referring expression resolution, with an emphasis on 2d/3d real world and map-based application. language grounding refers to interpreting language in a situated context and includes collaborative language grounding toward situated humanrobot dialog @xcite , city exploration @xcite , as well as following high-level navigation instructions @xcite . mapping instructions to low level actions has been explored in structured environments by mapping raw visual representations of the world and text onto actions using using reinforcement learning methods @xcite @xcite . this work has recently been extended to controlling autonomous systems and robots through human language instruction in a 3d simulated environment @xcite @xcite and mixed reality @xcite and using imitation learning @xcite . these systems perform goal prediction and action generation to control a single unmanned aerial vehicles (uavs), given a natural language instruction, a world representation and/or robot observations. however, where this prior work uses raw pixels to generate a persistent semantic map from the system's line-of-sight image, our model is able to leverage both pixel and metadata, when it is available in a combined approach. other approaches include neural mapping of navigational instructions to action sequences @xcite , which does include a representation of the observable world state, but this is more akin to a maze rather than a complex map. with respect to the task, our model looks to predict gps locations. there are few related works that attempt this challenging task. one study, as part of the ecml/pkdd challenge @xcite , uses neural networks for taxi destination prediction as a sequence of gps points. however, this does not include processing natural language instructions. spaceref @xcite is perhaps the closest to our task in that the task entails both gps tracks in osm and annotated mentions of spatial entities in natural language. however, it is different in that these spatial entities are viewed and referred to in a first person view, rather than entities on a map (e.g. "the arch at the bottom"). in terms of our choice of model, attention mechanisms @xcite @xcite have proven to be very powerful in language and vision tasks and we draw inspiration from the way @xcite use attention to solve image captioning by associating words to spatial regions within a given image. ## data as mentioned above, the task is based on open-streetmap (osm) @xcite . osm is a massively collaborative project, started in 2004, with the main goal to create a free editable map of the world. the data is available under the open data commons open database licence and has been used in some prior work @xcite @xcite . it is a collection of publicly available geodata that are constantly updated by the public and consists of many layers of various geographic attributes of the world. physical features such as roads or buildings are represented using tags (metadata) that are attached to its basic data structures. a comprehensive list of all the possible features available as metadata can be found online 2 . there are two types of objects, nodes and ways, with unique ids that are described by their latitude/longitude (lat/lon) coordinates. nodes are single points (e.g. coffee shops) whereas ways can be more complex structures, such as polygons or lines (e.g. streets and rivers). for this study, we train and test only on data that uses single points (nodes) and polygons (using the centre point), and leave understanding more complex structures as future work. there are three linguistic phenomena of note that we observe in the data collected. firstly, landmark grounding where each scenario has 3-5 generated robots and an average of 30 landmarks taken from osm. each subject could refer to any of these objects on the map, in order to complete the task. grounding the right noun phrase to the right osm landmark or robot, is crucial for predicting accurately the gold-standard coordinate, e.g. send husky11 62m to the west direction or send 2 drones near harborside park. secondly, bearing/distance factors need to be extracted from the instruction such as numbers (e.g. 500 meters) and directions (e.g. northwest, ne) and these two items typically come together. for example, "send drone11 to the west about 88m". thirdly, spatial relations are where prepositions are used instead of distance/bearing (e.g. near, between), and are thus more vague. for example, "send a drone near the silver strand preserve". ## results ## conclusion and future work we have developed a model that is able to process instructions on a map using metadata from rich map resources such as osm and can do so for maps that it has not seen before with only a 10% reduction in accuracy. if no metadata is available then the model can use vision, although this is clearly a harder task. vision does seem to help in examples where there is a level of uncertainty such as with spatial relations or ambiguity between entities. future work will involve exploring this further by training the model on these type of instructions and on metadata that are scarce and inaccurate. finally, these instructions will be used in an end-to-end dialog system for remote robot planning, whereby multi-turn interaction can handle ambiguity and ensure reliable and safe destination prediction before instructing remote operations.