Into the Unknown: Generating Geospatial Descriptions for New Environments (2024)

Abstract: Similar to vision-and-language navigation (VLN) tasks that focus on bridging the gap between vision and language for embodied navigation, the new Rendezvous (RVS) task requires reasoning over allocentric spatial relationships using non-sequential navigation instructions and maps. However, performance substantially drops in new environments with no training data.Using opensource descriptions paired with coordinates (e.g., Wikipedia) provides training data but suffers from limited spatially-oriented text resulting in low geolocation resolution. We propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. Our method constructs a grounded knowledge-graph, capturing entity relationships. Sampled entities and relations (“shop north of school”) generate navigation instructions via (i) generating numerous templates using context-free grammar (CFG) to embed specific entities and relations; (ii) feeding the entities and relation into a large language model (LLM) for instruction generation. A comprehensive evaluation on RVS, showed that our approach improves the 100-meter accuracy by 45.83% on unseen environments. Furthermore, we demonstrate that models trained with CFG-based augmentation achieve superior performance compared with those trained with LLM-based augmentation, both in unseen and seen environments. These findings suggest that the potential advantages of explicitly structuring spatial information for text-based geospatial reasoning in previously unknown, can unlock data-scarce scenarios.

## introduction the ability to extract locations and paths from natural language descriptions of spatial information holds immense significance. this capability proves crucial in daily and disaster response scenarios, aiding the billions globally lacking formal addresses @xcite @xcite , and enhancing geographic information retrieval (gir), particularly leveraging web-based resources @xcite . echoing the vision-and-language navigation (vln) task's goal, of bridging the gap between visual perception and natural language (nl) instructions for embodied agents @xcite , the recently introduced rendezvous (rvs) navigation task @xcite seeks to achieve a similar connection, but specifically be-tween map representations and natural language nl. while vln focuses on navigating within an environment based on visual cues and sequential instructions, rvs emphasizes the ability to utilize map information and non-sequential, often allocentric language descriptions to reach a specific target location. this shift from vision-centric instructions to map-aided guidance presents unique challenges, including reasoning about multiple spatial relationships simultaneously, inferring implicit actions from language, and navigating without explicit verification or step-by-step instructions. however, there is a substantial gap between current models and the human performance on the rvs task, particularly in new environments that lack human annotated data. one approach to address this issue is to leverage naturally-occurring open-source data such as wikipedia. however, these sources lack direct spatial information, which can result in models' low performance @xcite . synthesizing data using large language models (llms) is a common method for addressing data scarcity in nlp @xcite . however, for multimodal scenarios requiring precise spatial relationships, accurately generating such data without introducing errors or "hallucinations" (i.e., spurious relationships or entities), which severely undermines the performance on the underlying downstream task, remains a significant challenge. we propose a method for generating high-quality synthetic data for new environments using opensource geospatial data (figure 1 ). our method constructs a grounded knowledge-graph of the environment, capturing spatial relationships between entities. by sampling these entities, abstract shapes like 'blocks' (implicitly derived from street relationships), and relations (e.g., 'the garden is next to a restaurant'), we generate navigation instructions by either (i) creating a large amount of templates via a generative context-free grammar (cfg), in which we embed the precise entities and relations sampled, or (ii) feeding the entities and relation into an llm which generates the instructions. extensive evaluation on the rvs dataset demonstrates the clear advantage of our cfg-based method compared to the llm approach. when navigating unseen environments, our method achieves a remarkable 9.1% absolute increase in 100-meter accuracy and a substantial 39-meter decrease in median distance error. overall, our method helps close the human-ai performance gap in unseen environments by 45.83% in 100m accuracy and a decrease of 1,183m in median distance error. for the seen environment, our method results in an absolute improvement of 19.56% in 100m accuracy and a decrease of 151m in median distance error. ## the task the rendezvous (rvs, paz-argaman et al., 2024) task evaluates a system's ability to follow humangenerated, colloquial language navigation instructions within a dense urban environment depicted by a map. the system is provided with three inputs: (i) detailed map as knowledge graph: a comprehensive map of the environment represented as a knowledge graph. this graph encodes spatial relationships between landmarks and other relevant features. (ii) explicit starting point (geo-coordinates): the starting location specified as a latitude and longitude coordinate pair. (iii) navigation instruction: a natural language instruction describing the target location's relative position to landmarks and the starting point. this instruction leverages colloquial language typically used in navigation scenarios. the rvs task demands the system to process these inputs and generate the goal location's coordinates within the defined map boundaries. ## proposed: relational augmentation our augmentation method aims to generate natural language location descriptions that are both accurate and well grounded. to that end, we leverage openstreetmap (osm). ## background and related work text-based navigation tasks constitute a multimodal challenge @xcite @xcite demanding the integration of language comprehension and environmental knowledge. this environment can be either indoor, commonly represented through the agent's visual perception as it navigates @xcite , or outdoor. for outdoor environments, representation falls into three main categories: (i) visual representation -similar to indoor settings, real-world imagery can be employed, with agents learning the environment through street-view like exploration @xcite ; (ii) map-based representation -agents navigate based on map perception, as seen in studies utilizing maps as the primary input @xcite ; and (iii) hybrid representation -a combination of visual and map information @xcite . realistically, we would like to learn to navigate based on text in new environments, ones that our models did not train on before. furthermore, environments constantly change, such that we need to reacquaint our models with these changes in order for them to stay relevant @xcite . synthetic data generation, facilitated by llms, has become a prominent approach in nlp to mitigate data @xcite . this technique is particularly attractive due to its adaptability to various tasks and its ability to generate substantial volumes of data, allowing for robust model training and improved performance. data generation has also been applied to multimodal tasks, demonstrating performance close to that of human-annotated data @xcite . previous works on vision and language navigation tasks (vln) tried to tackle the data scarcity issue by generating synthetic data with a generative model trained with human annotated data @xcite @xcite . however, the learned distribution was limited to the environment the model was trained on. envedit @xcite tackled the vln task by generating new environments and synthetic navigation instructions in order to teach models to generalize to new environments. the new environments created differ in style, appearance, or the configuration of the objects. however, a lack of new objects hinders unseen object handling. @xcite leveraged image-to-image synthesis via a generative adversarial network (gan) architecture @xcite to create new viewpoints for existing environments and subsequently generate synthetic instructions for these environments. ## conclusion this work presents a novel data-augmentation solution for spatial nlp tasks. leveraging spatial relation extraction it generates a vast, albeit slightly less refined, dataset compared to human annotations. this quantitative advantage unlocks superior performance, as evidenced by a 44.54% absolute improvement in 100m accuracy and a 1,170m reduction in median absolute error distance on unseen environments in the rvs dataset. these results demonstrate the effectiveness of our approach in handling novel environments with sparse or no human data. moreover, our cfg-based augmentation method offers adherence to correct spatial relations, control over the content, and interpretability, over llms in spatial descriptions and could serve as a future tool for evaluation, detection, and mitigation of artifacts such as 'hallucinations'.