G eo LM : Empowering Language Models for Geospatially Grounded Language Understanding (2023)

Abstract: Humans subconsciously engage in geospatial reasoning when reading articles. We recognize place names and their spatial relations in text and mentally associate them with their physical locations on Earth. Although pretrained language models can mimic this cognitive process using linguistic context, they do not utilize valuable geospatial information in large, widely available geographical databases, e.g., OpenStreetMap. This paper introduces GeoLM, a geospatially grounded language model that enhances the understanding of geo-entities in natural language. GeoLM leverages geo-entity mentions as anchors to connect linguistic information in text corpora with geospatial information extracted from geographical databases. GeoLM connects the two types of context through contrastive learning and masked language modeling. It also incorporates a spatial coordinate embedding mechanism to encode distance and direction relations to capture geospatial context. In the experiment, we demonstrate that GeoLM exhibits promising capabilities in supporting toponym recognition, toponym linking, relation extraction, and geo-entity typing, which bridge the gap between natural language processing and geospatial sciences. The code is publicly available at https://github.com/knowledge-computing/geolm.

## introduction spatial reasoning and semantic understanding of natural language text arise from human communications. for example, "i visited paris in arkansas to see the smaller eiffel tower", a human can easily recognize the toponyms "eiffel tower", "paris", "arkansas", and the spatial relation "in". implicitly, a human can also infer that this "eiffel tower" might be a replica foot_0 of the original one in paris, france. concretely, the process of geospatially grounded language understanding involves core tasks such as recognizing geospatial concepts being described, inferring the identities of those concepts, and reasoning about their spatially qualified relations. these tasks are essential for applications that involve the use of place names and their spatial relations, such as social media message analysis @xcite , emergency response @xcite , natural disaster analysis @xcite , and geographic information retrieval @xcite . pretrained language models (plms; @xcite @xcite ) have seen broad adaptation across various domains such as biology @xcite , healthcare @xcite , law @xcite , software engineering @xcite , and social media @xcite . these models benefit from in-domain corpora (e.g., pubmed for the biomedical domain) to learn domain-specific terms and concepts. similarly, a geospatially grounded language model requires training with geo-corpora. the geo-corpora should cover worldwide geo-entity names and their variations, geographical locations, and spatial relations to other geo-entities. although some geo-corpora are available, e.g., lgl @xcite and spa-tialml @xcite , the sizes of the datasets are relatively small. in contrast, wikipedia stores many articles describing places worldwide and can serve as comprehensive geo-corpora for geospatially grounded language understanding. however, training with wikipedia only solves partial challenges in geospatial grounding, as it only provides the linguistic context of a geo-entity with sentences describing history, demographics, climate, etc. the information about the geospatial neighbors of a geoentity is still missing. on the other hand, large-figure 1 : outline of geolm. wikipedia and wikidata form the nl corpora, and openstreetmap (osm) form the pseudo-sentence corpora (details in §2.2). geolm takes the nl and pseudo-sentence corpora as input, then pretrain with mlm and contrastive loss using geo-entities as anchors. (see §2.3) scale geographical databases (e.g., openstreetmap) and knowledge bases (e.g., wikidata) can provide extensive amounts of geo-entity locations and geospatial relations, enriching the information sourced from wikipedia. additionally, the geolocations from openstreetmap can help connecting the learned geo-entity representations to physical locations on the earth. to address these challenges, we propose ge-olm, a language model specifically designed to support geospatially grounded natural language understanding. our model aims to enhance the comprehension of places, types, and spatial relations in natural language. we use wikipedia, wikidata, and osm together to create geospatially grounded training samples using geo-entity names as anchors. geolm verbalizes the wikidata relations into natural language and aligns the linguistic context in wikipedia and wikidata with the geospatial context in osm. since existing language models do not take geocoordinates as input, geolm further incorporates a spatial coordinate embedding module to learn the distance and directional relations between geo-entities. during inference, our model can take either natural language or geographical subregion (i.e., a set of nearby geo-entities) as input and make inferences relying on the aligned linguistic-geospatial information. we employ two training strategies: contrastive learning @xcite between natural language corpus and linearized geographic data, and masked language modeling @xcite with a concatenation of these two modalities. we treat geolm as a versatile model capable of addressing various geographically related language understanding tasks, such as toponym recognition, toponym linking, and geospatial relation extraction. ## geolm this section introduces geolm's mechanism for representing both the linguistic and geospatial context ( §2.1), followed by the detailed development process of pretraining tasks ( §2.3) and corpora ( §2.2), and how geolm is further adapted to various downstream geospatial nlu tasks ( §2.4). ## experiments we hereby evaluate geolm on the aforementioned four downstream tasks. all compared models (except gpt3.5) are finetuned on task-specific datasets for toponym recognition, geo-entity typing, and geospatial relation extraction. 4 4 toponym linking is an unsupervised task. ## conclusion in this paper, we propose geolm, a plm for geospatially grounded language understanding. this model can handle both natural language inputs and geospatial data inputs, and provide connected representation for both modalities. technically, geolm conducts contrastive and mlm pretraining on a massive collection of distantly parallel language and geospatial corpora, and incorporates a geocoordinate embedding mechanism for improved spatial characterization. through evaluations on four important downstream tasks, toponym recognition, toponym linking, geo-entity typing and geospatial relation extraction, geolm has demonstrated competent and comprehensive abilities for geospatially grounded nlu. in the future, we plan to extend the work to question answering and autoregressive tasks. li yu and feng lu. 2015. a bootstrapping algorithm for geo-entity relation extraction from online encyclopedia. in 2015 23rd international conference on geoinformatics, pages 1-5. ieee.