S pa BERT : A Pretrained Language Model from Geographic Data for Geo-Entity Representation (2022)

Abstract: Named geographic entities (geo-entities for short) are the building blocks of many geographic datasets. Characterizing geo-entities is integral to various application domains, such as geo-intelligence and map comprehension, while a key challenge is to capture the spatial-varying context of an entity. We hypothesize that we shall know the characteristics of a geo-entity by its surrounding entities, similar to knowing word meanings by their linguistic context. Accordingly, we propose a novel spatial language model, SpaBERT, which provides a general-purpose geo-entity representation based on neighboring entities in geospatial data. SpaBERT extends BERT to capture linearized spatial context, while incorporating a spatial coordinate embedding mechanism to preserve spatial relations of entities in the 2-dimensional space. SpaBERT is pretrained with masked language modeling and masked entity prediction tasks to learn spatial dependencies. We apply SpaBERT to two downstream tasks: geo-entity typing and geo-entity linking. Compared with the existing language models that do not use spatial context, SpaBERT shows significant performance improvement on both tasks. We also analyze the entity representation from SpaBERT in various settings and the effect of spatial coordinate embedding.

## introduction interpreting human behaviors requires considering human activities and their surrounding environment. looking at a stopping location, [ speedway , ], foot_0 from a person's trajectory, we might assume that this person needs to use the location's amenities if speedway implies a gas station, and is near a highway exit. we might predict a meetup at @xcite if the trajectory travels through many other locations, , , ..., of the same name, speedway , to arrive at [ speedway , ] in the middle of farmlands. as humans, we are able to make such inferences using the name of a geographic entity (geo-entity) and other entities in a spatial neighborhood. specifically, we contextualize a geo-entity by a reasonable surrounding neighborhood learned from experience and, from the neighborhood, relate other relevant geo-entities based on their name and spatial relations (e.g., distance) to the geo-entity. this way, even if two gas stations have the same name (e.g., speedway ) and entity type (e.g., 'gas station'), we can still reason about their spatially varying semantics and use the semantics for prediction. capturing this spatially varying location semantics can help recognizing and resolving geospatial concepts (e.g., toponym detection, typing and linking) and the grounding of geo-entities in documents, scanned historical maps, and a variety of knowledge bases, such as wikidata, open-streetmap, and geonames. also, the location semantics can support effective use of spatial textual information (geo-entities names) in many spatial computing task, including moving behavior detection from visiting locations of trajectories @xcite , point of interest recommendations @xcite , air quality @xcite @xcite and traffic prediction @xcite using location context. recently, the research community has seen a rapid advancement in pretrained language models (plms) @xcite @xcite , which supports strong contextualized language representation abilities @xcite and serves as the backbones of various nlp systems @xcite . the extensions of these plms help nl tasks in different data domains (e.g., biomedicine @xcite , software engineering @xcite , finance @xcite ) and modalities (e.g., tables @xcite @xcite and images @xcite ). however, it is challenging to adopt existing plms or their extensions to capture geo-entities' spatially varying semantics. first, geo-entities exist in the physical world. their spatial relations (i.e., distance and orientation) do not have a fixed structure (e.g., within a table or along a row of a table) that can help contextualization. second, existing language models (lms) pretrained on general domain corpora @xcite require fine-tuning for domain adaptation to handle names of geo-entities. to tackle these challenges, we present spabert ( ), a lm that captures the spatially varying semantics of geo-entity names using large geographic datasets for entity representation. built upon bert @xcite , spabert generates the contextualized representation for a geo-entity of interest (referred to as the pivot), based on its geographically nearby geoentities. specifically, spabert linearizes the 2-dimensional spatial context by forming pseudo sentences that consist of names of the pivot and neighboring geo-entities, ordered by their spatial distance to the pivot. spabert also encodes the spatial relations between the pivot and neighboring geo-entities with a continuous spatial coordinate embedding. the spatial coordinate embedding models horizontal and vertical distance relations separately and, in turn, can capture the orientation relations. these techniques make the inputs compatible with the bert-family structures. in addition, the backbone lm, bert, in spabert is pretrained with general-purpose corpora and would not work well directly on geoentity names because of the domain shift. we thus train spabert using pseudo sentences generated from large geographic datasets derived from openstreetmap (osm). 2 this pretraining process conducts masked language modeling (mlm) and masked entity prediction (mep) that randomly masks subtokens and full geo-entity names in the pseudo sentences, respectively. mlm and mep enable spabert to learn from pseudo sentences for generating spatially varying contextualized representations for geo-entities. spabert provides a general-purpose representation for geo-entities based on their spatial context. similar to linguistic context, spatial context refers to the surrounding environment of a geo-entity. for example, [ speedway , ], [ speedway , ], and [ speedway , ] would have different representations since the surrounding environment of , , and could vary. we evaluate spabert on two tasks: 1) geo-entity typing and 2) geo-entity linking to external knowledge bases. our analysis includes the performance comparison of spabert in various settings due to characteristics of geographic data sources (e.g., entity omission). to summarize, this work has the following contributions. we propose an approach to linearize the 2-dimensional spatial context, encode the geoentity spatial relations, and use a lm to produce spatial varying feature representations of geoentities. we show that spabert is a generalpurpose encoder by supporting geo-entity typing and geo-entity linking, which are keys to effectively integrating large varieties of geographic data sources, the grounding of geo-entities as well as supports for a broad range of spatial computing applications. the experiments demonstrate that spabert is effective for both tasks and outperforms the sota lms. ## spabert spabert is a lm built upon a pretrained bert and further trained to produce contextualized geo-entity representations given large geographic datasets. fig. 1 shows the outline of spabert, with the details below. this section first presents the preliminary ( §2.1) and then describes the overall approach for learning geo-entities' contextualized representations ( §2.2), the pretraining strategies ( §2.3), and inference procedures ( §2.4). ## experiments we evaluate spabert on supervised geo-entity typing and unsupervised geo-entity linking tasks ( §3.1- §3.3). we also investigate how spabert performs under various common characteristics of geographic data sources (e.g., entity omission from cartographic generalization) and how critical technical components and their parameters affect the performance ( §3.4). ## related work pretrained language models plms have been the dominant paradigm for language representation. following the success of mlms @xcite and autoregressive plms @xcite , more recent work has extended the pretraining process with more tasks or learning objectives for span prediction @xcite , cross-encoders (reimers and gurevych, 2019), contrastive learning @xcite , and massively multi-task learning @xcite . to support entity representation, several approaches propose to perform mention detection @xcite , incorporate mention memory cells @xcite , or injecting structural knowledge representations @xcite @xcite . due to the large body of work, we refer readers to recent surveys summarizing this line of work @xcite . to extend the use of plms beyond language, much exploration has also been conducted to represent other modalities. for example, a significant amount of vision-language models @xcite @xcite have been developed by jointly pretraining on co-occurring vision and language corpora, and are in the support of grounding @xcite , generation @xcite and retrieval tasks @xcite on the vision modality. other plms capture semi-structure tabular data by linearizing table cells @xcite @xcite or incorporating structural prior in the attention mechanism @xcite @xcite . to the best of our knowledge, none of the prior studies have extended pretrained lms to geo-entity representation, nor do they support a suitable mechanism to capture the geo-entities' spatial relations, which do not have structural prior. this is exactly the focus of our work. ## conclusion this paper presented spabert ( ), a language model trained on geographic datasets for contextualizing geo-entities. spabert utilizes a novel spatial coordinate embedding mechanism to capture spatial relations between 2d geo-entities and linearizes the geo-entities into 1d sequences, compatible with the bert-family structures. the experiments show that the general-purpose representations learned from spabert achieve better or competitive results on the geo-entity typing and geo-entity linking tasks compared to sota pretrained lms. we plan to evaluate spabert on additional related tasks, such as geo-entity to natural language grounding. also, we plan to extend spabert to support other geo-entity geometry types, including lines and polygons.