Text-to- O verpass QL : A Natural Language Interface for Complex Geodata Querying of O pen S treet M ap (2024)

Abstract: We present Text-to-OverpassQL, a task designed to facilitate a natural language interface for querying geodata from OpenStreetMap (OSM). The Overpass Query Language (OverpassQL) allows users to formulate complex database queries and is widely adopted in the OSM ecosystem. Generating Overpass queries from natural language input serves multiple use-cases. It enables novice users to utilize OverpassQL without prior knowledge, assists experienced users with crafting advanced queries, and enables tool-augmented large language models to access information stored in the OSM database. In order to assess the performance of current sequence generation models on this task, we propose OverpassNL,1 a dataset of 8,352 queries with corresponding natural language inputs. We further introduce task specific evaluation metrics and ground the evaluation of the Text-to-OverpassQL task by executing the queries against the OSM database. We establish strong baselines by finetuning sequence-to-sequence models and adapting large language models with in-context examples. The detailed evaluation reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.

## introduction the openstreetmap (osm) database stores vast amounts of structured knowledge about our world. users mainly access it via applications that render a visual map of inquired areas. a more advanced and systematic way to examine the stored information is to query the underlying geodata using * equal contribution. ♦ work done while at computational linguistics, heidelberg university. the overpass query language (overpassql). overpassql is a feature-rich query language that is widely adopted in the osm ecosystem by contributors, analysts, and applications. in order to make this empowering query language accessible via natural language, we propose the textto-overpassql task. the objective is to take a complex data request in natural language and translate it to overpassql in order to execute it against the osm database. several groups of users can benefit from such a natural language interface. inexperienced users are spared from learning the overpassql syntax. expert users can use it to draft overpass queries and then manually refine them, saving time and mental load in comparison to writing the full query from scratch. another use-case is to incorporate the interface as an api tool @xcite for large language models (llms). in this work, we introduce the three components that enable the text-to-overpassql task. first, we present overpassnl, a dataset of 8.5k natural language inputs and corresponding overpass queries. the queries were collected from an osm community website where they were written by osm users to fulfill legitimate information needs. we then hired and trained students to write natural language descriptions of the queries. second, we introduce a systematic evaluation protocol that assesses the prediction quality of a candidate system. to this end, we propose a task-specific metric that takes the similarity of the system output to overpass queries on the levels of surface string, semantics, and syntax into account. moreover, we ground the evaluation by executing the generated query against the osm database and compare the returned elements with those returned by the gold query. third, we explore several models and learning strategies to establish a base performance for the problem of generating figure 1 : natural language input and the corresponding overpass query. the query is executed against the openstreetmap database and returns the requested elements in a structured response. the overpass query language is highly expressive and allows one to formulate complex queries to extract information from openstreetmap. blue tokens in the query are syntax keywords, orange tokens are variable names, and bold tokens define semantic properties of the requested elements. the green token in curly brackets geolocates an area called ''troms''. overpass queries from natural language. we finetuned sequence-to-sequence models and found that explicitly pretraining on code is not helpful for the task. we further explored in-context learning strategies for black-box llms and found that gpt-4 (openai, 2023) with few-shot examples, retrieved by sentence similarity, yields best results, outperforming the finetuned sequence-tosequence models. the proposed text-to-overpassql task, depicted in figure 1 , is a novel semantic parsing problem that is well motivated in real-world applications. while it shares characteristics with the text-to-sql task and its accompanying datasets, there are several key differences. the text-to-overpassql task is grounded in a database that is genuinely in use and of global scale. the database is not divided into sub-databases or tables, and each overpass query can retrieve any of the billions of stored elements in osm. the desired elements have to be queried by a geographical specification and additional semantic tags. the tags are composed of key-value pairs that follow established community guidelines and conven-tions, but can also be open-vocabulary. overall, the proposed task builds on a decades-long effort to structure and store the geographical world around us to make it computationally accessible. with this work, we offer all components to benchmark future semantic parsing systems on a challenging real-world task. our main contributions are as follows: (i) we present overpassnl, a dataset of 8.5k natural language inputs paired with real-world overpass queries. (ii) we define task-specific evaluation metrics that take the overpassql syntax into account and are grounded in database execution. (iii) we train and evaluate several state-of-the-art sequence generation models to establish base performance and to identify specific properties of the proposed text-to-overpassql task. ## overpassnl dataset in order to facilitate the text-to-overpassql task, we constructed a parallel dataset of natural language inputs and overpass queries. this was done by collecting queries written and shared by users of overpass turbo, foot_3 a web tool that allows users to develop and execute overpass queries within a graphical interface. we presented the queries to trained annotators who were tasked with writing natural language descriptions for them. initiating the dataset creation process with overpass queries authored by real users and developers has several advantages: firstly, the queries were created to satisfy legitimate information needs and, as such, cover a wide range of over-passql features. also, the geographical coverage is high, as is shown in figure 3 by the location distribution of elements returned by executing the queries in our dataset. another advantage is that it is easier to teach annotators how to interpret overpass queries than how to write them from scratch. ## task & evaluation the text-to-overpassql task requires a user to generate an overpass query q, given a natural language input x. a model for this task aims to accurately translate the request, formulated in natural language, into code, written in the overpass query language, that returns the correct elements when executed against the openstreet-map database. in order to evaluate such a system, we propose to use different evaluation metrics. these include metrics that compare the generated query with the reference query, and metrics that compare the results returned by executing the queries against the osm database. in order to make the execution results reproducible, we release the evaluation script and a docker container with the exact snapshot of the osm database we used. we further describe the metrics in detail. ## experiments in the following, we present experiments that showcase the opportunities to train machine learning models on the overpassnl dataset and establish base performance of commonly used techniques. we finetuned sequence-to-sequence models of different sizes and pretraining settings. additionally, we adapted black-box large language models with different in-context learning strategies. all models are evaluated with the proposed similarity and grounded metrics, as well as exact string match (em). ## analysis 7.1 instance difficulty to better assess the performance of our proposed models, we aim to divide the evaluation instances into three difficulty partitions. figure 6 displays ex soft results for the easy, medium, and hard partitions according to different difficulty criteria. a straightforward difficulty metric like the length of the input text has little influence on the accuracy. using the length or complexity of the query (measured as the number of syntactic units) as difficulty metric leads to a clearer partition into easy, medium, and hard instances. surprisingly, partitioning the instances based on their maximum input text similarity to any training instance leads to an undesired negative correlation of ex soft and difficulty where the instances with the lowest similarity to training inputs achieve best ex soft results. finally, the clearest partition into instances of different difficulty is achieved by using the maximum similarity of a query to any query in the training set, where query similarity is measured by the oqs metric. a partitioning based on similarity to training queries can be seen as out-of-distribution testing since it measures the performance for instances that are less likely to be memorized from the training set. we thus use this criterion to select instances constituting the hard partition in the experiments in section 6. ## conclusion we introduced a novel semantic parsing task, called text-to-overpassql. the objective of this task is to generate overpass queries from natural language inputs. we highlighted its relevance within the openstreetmap ecosystem and related it to similar tasks. we identified key differences to the text-to-sql that pose unique challenges. to facilitate research on the task, we proposed over-passnl, a dataset of real-world overpass queries and corresponding annotations with natural language inputs. we used this dataset to train several state-of-the-art models and establish a base performance of the task. in order to measure prediction performance, we proposed task-specific metrics that take the overpassql syntax into account and are grounded in database execution. we presented a detailed evaluation of results that reveals the strengths and weaknesses of the considered learning strategies. we hope that our work serves as a foundation for further research on the challenging task of semantic parsing of geographical information needs grounded in the large and widely used openstreetmap database.