A Dataset and Evaluation Framework for Complex Geographical Description Parsing (2020)

Abstract: Much previous work on geoparsing has focused on identifying and resolving individual toponyms in text like Adrano, S.Maria di Licodia or Catania. However, geographical locations occur not only as individual toponyms, but also as compositions of reference geolocations joined and modified by connectives, e.g., “. . . between the towns of Adrano and S.Maria di Licodia, 32 kilometres northwest of Catania”. Ideally, a geoparser should be able to take such text, and the geographical shapes of the toponyms referenced within it, and parse these into a geographical shape, formed by a set of coordinates, that represents the location described. But creating a dataset for this complex geoparsing task is difficult and, if done manually, would require a huge amount of effort to annotate the geographical shapes of not only the geolocation described but also the reference toponyms. We present an approach that automates most of the process by combining Wikipedia and OpenStreetMap. As a result, we have gathered a collection of 360,187 uncurated complex geolocation descriptions, from which we have manually curated 1,000 examples intended to be used as a test set. To accompany the data, we define a new geoparsing evaluation framework along with a scoring methodology and a set of baselines.

## introduction geoparsing, or toponym resolution, is the task of identifying geographical entities, and attaching them to their corresponding reference in a coordinate system @xcite . in its traditional setting it figure 1: an illustrative example of complex geographical description parsing. given a text describing a target geolocation the goal is to aproximate its geometry using as reference the geometries of the geolocations that appear in the description (adrano, s.maria di licodia and catania). in this example, the target geolocation is biancavilla, but in general, there may not be a name for the target geolocation. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . the task plays a key role in information extraction, since it allows events to be placed in an unequivocal location that could be plotted on a map. however, the setting described above restricts the task to geolocations with explicit names, whilst text in natural language can refer to geographical regions by complex descriptions that involve a set of different toponyms. take, for example, the following snippet: "a town and comune in the metropolitan city of catania, sicily, southern italy. . . located between the towns of adrano and s.maria di licodia, 32 kilometres (20 mi) northwest of catania.". traditional geoparsing could return coordinates for the individual toponyms (adrano, s.maria di licodia, etc.) but not the boundaries of the geolocation described by the entire phrase. solving this problem requires understanding the meaning of linguistic connectives (e.g. between, northwest) and a geometric interpretation of how they combine the geographical entities. these complex descriptions are frequent in domains where there are no specific names for geographical regions of interest, for example in environmental impact statements describing the effects on the environment of activities such as constructing new mines or roads, or in food security outlooks describing the impact of localized agricultural production and other factors on humanitarian crises. though parsing these kinds of geographical descriptions and aproximating the corresponding geometries has been the goal of some previous work @xcite , research in this area has been constrained by the lack of a benchmark dataset. a naive approach to building such a dataset would require a huge amount of resources to find and collect complex geograhical descriptions, to annotate the reference toponyms and to edit the geometries that correspond to the target locations. in this paper, we present a methodology to acquire such a dataset by combining wikipedia and openstreetmap foot_1 , making use of the links between those resources. we collect descriptions of geolocations from wikipedia articles that are linked to an openstreetmap geometry, i.e. the set of coordinates that define the boundary of a geographical location. the text from the wikipedia pages retrieved includes embedded links to other geolocations for which openstreetmap geometries can be obtained. for example, the snippet presented above corresponds to the description of biancavilla in wikipedia. in our proposed task, we assume that the geometry of biancavilla is unknown and must be recovered from its description using the rest of toponoym geometries (adrano, s. maria di licodia, . . . ) as reference, as shown in figure 1 . as the content of both wikipedia and openstreetmap has been manually created and reviewed by a broad commnunity of volunteers, the remaining steps needed for dataset construction are to collect the descriptions, filter non-compositional examples, and, in a some cases, complete missing reference geolocations. in summary, the main contributions of this paper are the following: the dataset and evaluation framework are available at https://github.com/egolaparra/ geocode-data and https://github.com/egolaparra/geocode respectively. ## task description before going into further details about the building of our dataset, we first formally define the task of parsing complex geographical descriptions. formally, a geoparser for such descriptions is a function h : (t, l) → g, where: intuitively, the goal is to be able to map a natural language description and the geographical regions it mentions to a new geographical region that represents the complex geolocation described. all geographical regions are defined in terms of longitude and latitude points in the world geodetic system (wgs 84) @xcite foot_2 . this task is challenging because every linguistic composition of phrases must be matched to a geometric composition of geographic regions. consider the first part of the description in figure 1 : "between the towns of adrano and s.maria di licodia". to parse this expression, one must first identify the phrases that represent spatial concepts (e.g. between, adrano, s.maria di licodia). phrases that represent known geographical regions must then be retrieved from an index of such geometries. for example, geometry(adrano) = {(14.8258023, 37.6324199), (14.8275266, 37.6319645), . . .} in openstreetmap. phrases that represent compositions of other geographical regions in the description must be mapped to formal geometric operations. for example, between should be mapped to: ## related work the task proposed in this paper can be seen as a junction of two different nlp research lines. on the one hand, toponym resolution has been widely studied, especially within the entity linking framework @xcite . but it has gained interest as an independent task in recent years. its ultimate goal is to retrieve the coordinates for the geolocation mentioned in the text, for which gazetteers, such as geonames and wikipedia, are frequently used as reference knowledge bases. the problem is commonly separated into two steps. first, location references are extracted from the text following a named entity recognition strategy @xcite . secondly, the references are disambiguated and linked to the reference knowledge base @xcite @xcite . as pointed out by @xcite , the amount of annotated data for toponym resolution is not large, but a few datasets are available for different domains such as historical texts @xcite , social media @xcite , scientific literature @xcite and wikipedia @xcite . thanks to these works, it is possible to link locations mentioned in text to unambiguous geographic respresentations, in the form of coordinates. however, they do not address the compositional nature of the geographical descriptions. on the other hand, although not necesarily related to toponyms, previous works have studied the structure of (geo)spatial natural language expressions. however, many of these works limited their attention to the role of specific elements, such as prepositions @xcite @xcite , or particular grammatical structures @xcite . some other works defined the task as a tagging problem, each specifying a different set of labels to tag the components of the text. these works were focused on the relations between the elements of the geospatial expressions in a syntactically inspired manner @xcite @xcite . as pointed out by @xcite , the manual annotation of compositional (geo)spatial expressions is very challenging and, in consequence, many of these schemas are limited to a reduced set of elements or define complex tag structures. moreover, the goal of these works is to obtain a tag structure on top of the text but they do not provide an explicit way to intepret them to obtain a single geographical object, i.e. it is not possible to translate these structures into a set of coordinates. our work takes a different approach by not imposing any annotation structure on top of the text and providing, instead, the target and reference geometric shapes, in wgs84 coordinates, of the geolocation descriptions. by doing so, we aim to merge benefits from both research lines. ## dataset building we constructed a dataset from wikipedia, wikidata and openstreetmap as follows: figure 3 : the geometries retrieved from openstreetmap for the geolocations in the example of figure 2 . each geometry is defined by the set of coordinates that makes its boundary. the openstreetmap geometries are defined by sets of coordinates and can be nodes (a single coordinate), ways (a collection of nodes forming a linestring or a polygon) and relations (a collection of any kind of entities). figure 3 presents the plotted maps for some examples. ## description of the resource the set of descriptions obtained by the procedure described in section 4 are stored in a collection of xml files. for every geolocation we include the cannonical wikipedia name as well as its corresponding identifier and type of entity in openstreetmap. along with the xml files, we built a postgresql database to store the geographic objects collected from openstreetmap. examples of the storage formats can be seen in appendix a.1. examples from the data described are visualized in figure 4 . the maps have been plotted using the geometries retrieved from openstreetmap. all the linestrings and polygons correspond to the geolocations, both target and references, from the descriptions below each map. finally, it should be noted that, although all the descriptions come from the same source, namely wikipedia, the resource covers a wide variety of locations, inluding landforms, political entities, human ## metrics as the main metric for models attempting this geoparsing task, we propose to measure the area of the predicted geometry that overlaps with the gold one. in other words, the metric calculates the area of the intersection between both geometries. this value can be obtained in precision and recall as follows: similarly, overall recall is defined as the average of inidividual recalls in o: note that the overlap based metric penalizes equally all predictions that do not intersect with a gold geometry. we propose an additional scoring method that gives some credit to predictions that are close to the gold geometries even if they do not overlap. for this, we extend the gold geometries by a scale factor of 2 and then calculate the precision of the predictions as explained previously. in other words, to calculate the precision p x2 , we double the size of gold geometries. similarly, we scale the predicted geometry by a factor of 2 and calculate the recall r x2 . this two complementary metrics can be applied with both strict and relaxed criteria. ## baselines in order to have a reference for future approaches, we propose four non-lingustic baselines that ignore the text description, and only execute straightforward combinations of the reference geolocations. models which do any real language processing of the text descriptions should be able to exceed these baselines. we also propose a linguistic baseline that constructs a semantic parser from a hand-assembled synchronous grammar to parse the description of the target geometry and produce a composition of operations over the reference geometries. we expect machine learning models that combine both text and reference geometries will outperform this baseline. ## conclusion in this paper, we presented a new dataset for complex geographical description parsing. this dataset aims to push the research in this area that has been constrained by the lack of a benchmark dataset. we detailed the methodology to adquire such a dataset by combining wikipedia and openstreetmap geometries, followed by a manual curation of 1,000 descriptions to be used as test set. we also described a novel scoring methodology that completes the evaluation framework. finally, we proposed a set of baselines, including a grammar-based semantic parser, to be used as reference by future works. the results show that this a very challenging task, but we expect that the limitations of a rule-based system can be addressed by more sophisticated models trained in the uncurated portion of our dataset.