Pretraining and Finetuning Language Models on Geospatial Networks for Accurate Address Matching (2024)

Abstract: We propose a novel framework for pretraining and fine-tuning language models with the goal of determining whether two addresses represent the same physical building. Address matching and building authoritative address catalogues are important to many applications and businesses, such as delivery services, online retail, emergency services, logistics, etc. We propose to view a collection of addresses as an address graph and curate inputs for language models by placing geospatially linked addresses in the same context. Our approach jointly integrates concepts from graph theory and weak supervision with address text and geospatial semantics. This integration enables us to generate informative and diverse address pairs, facilitating pretraining and fine-tuning in a self-supervised manner. Experiments and ablation studies on manually curated datasets and comparisons with state-of-the-art techniques demonstrate the efficacy of our approach. We achieve a 24.49% improvement in recall while maintaining 95% precision on average, in comparison to the current baseline across multiple geographies. Further, we deploy our proposed approach and show the positive impact of improving address matching on geocode learning.

## introduction entity matching (em) @xcite aims to identify and link various representations of the same real-world entities across multiple databases. em is a challenging task, particularly when entities are unstructured @xcite and of limited data quality i.e. there is lack of completeness and consistency in their descriptions. additionally, real-world em tasks @xcite often have limited labeled data and require significant labeling effort to develop accurate models. in this paper, we pose address * equal contribution matching as an em task to determine if two addresses represent the same physical building or not. addresses are important to many businesses, such as logistics, online retail, and emergency services, as they are the primary source of information used to determine the location. they exhibit variations in writing styles and patterns, resulting in considerable discrepancies across similar addresses and their components (e.g., building, road). it is common to provide colloquial addresses that use landmarks and other points-of-interest (poi) to denote the place. for example foot_0 , "abg bank, opp. network stone, mahapurii" and "plot no. 438 taj towers, abg bank, mahapuri" represent the same physical building but its hard to distinguish syntactically. further, neighbourhood provided by a customer can also be known by other vernacular names or be a part of a larger neighbourhood. these synonyms are often used interchangeably, making it challenging to comprehend the addresses. language models (lms) have become the defacto approach to model real-world text. however, most of the efforts focus on general domain corpora. recent studies @xcite @xcite show that domainspecific pretraining from scratch substantially outperforms continual pretraining of generic language models, thus demonstrating that the prevailing assumption in support of mixed-domain or general domain pretraining is not always applicable. pretraining is followed by finetuning that specializes lms by training it on in-domain dataset, but realworld data tends to be noisy. the lms need to be exposed to diverse and high-quality examples for finetuning a pretrained model effectively as they directly affect the model's ability to comprehend. lack of quality training data is a perennial problem @xcite for em. further, creating a representative training set for address matching is challenging for multiple reasons -(1) data distribution is heavily skewed towards negative pairs, i.e. no-match. (2) the average handle time for an annotator to label an address pair is three times higher on average when compared to other em tasks. (3) across addresses, it is very common that component values are vernacular, redundant, noisy, missing, or misspelled, thus leading to unstructured data problems. (4) considering the current trend towards employing language models (lms) for entity matching @xcite , utilizing a few thousand samples result in over-fitting @xcite the lms. this necessitates having a more sophisticated approach for address matching. in this paper, we tackle the above discussed challenges by proposing an effective strategy for pretraining and fine-tuning lms that incorporates real-world knowledge among addresses via geospatial semantics. given a corpus of addresses, we obtain links between addresses using historic delivery information and address text to create lm inputs by placing linked addresses in the same context window. our approach thus provides a natural fusion of language-based and graph-based self-supervised learning. our empirical evaluation shows significant improvements in pair-wise matching and geocode learning metrics compared to the existing baseline system and other state-ofthe-art systems. further, it should be noted that the structure of addresses are quite different for different geographies, hence the improvements observed across multiple geographies confirm the wide applicability and generic nature of our approach. in summary, our main contributions are -(1) we introduce neighbour relation prediction (nrp) training objective to pretrain lms that enables the model understand neighbourhood level nuances and align on the address structure. (2) our approach jointly integrates geospatial properties and address text with graph theory and weak supervision to curate diverse and informative address pairs to finetune the lms in a self-supervised manner. ## related work we can divide prior literature into three broad categories -rule-based, crowd-based and learning based solutions. rule-based solutions ei-ther rely on pre-defined matching rules such as dnf @xcite or dynamically synthesized em rules @xcite to find matching pairs. while rule-based solutions are highly interpretable, they are time and resource-intensive requiring domain experts to define the rules and may perform poorly on unstructured data @xcite . to alleviate these drawbacks, crowdbased solutions @xcite @xcite have been proposed that employ crowd-sourcing to manually identify matching tuples. however, such methods are time consuming and human labor cost is expensive which makes them not suitable for large scale real-world applications. recently (maheshwary and sohoney, 2023) leveraged active learning with graphs to improve matching performance for geospatial entities. currently, the state-of-the-art solutions for em now predominantly rely on deep learning or lm based approaches. ditto @xcite casts em as a sequence-pair classification problem based on finetuning pretrained lms across different domains. geobert @xcite integrate semantics and geographic information in the pre-trained representations of pois by mapping multiple geographic granularity into a unified latent space, to obtain the poi embeddings with geographic information. recently proposed, geoer @xcite includes a transformer block, a geocoding block, and a neighbourhood block and is widely used in wide variety of geospatial systems. ## methodology we present a self-supervised approach for pretraining and fine-tuning language models (lms) with the aim of internalising spatial knowledge into lms via geospatial semantics. instead of viewing the address corpus as a list of addresses, we view it as an address graph, where each node in the graph represents an address and edges between nodes capture spatial relevance between addresses. the edges of an address graph can be created using various techniques; in our case, we use historical delivery data to sample address pairs and assign spatial links based on the h3 geospatial indexing system @xcite for model pretraining. we also introduce the neighbour relation prediction (nrp) training objective to pretrain lms. this objective enables the model to understand neighbourhood-level nuances and align with the address structure. while fine-tuning, we generate spatial links among sample address pairs by integrating weak supervision with graph theory that leverages address text with historic delivery information. the intuition here is to let the model learn diverse variations across similar physical buildings within a neighbourhood. the workflow of our proposed framework is demonstrated in figure 1 and discussed in section 3.2 and 3.3. ## experiments we did extensive offline experimentation to develop, refine, and validate our approach. in this section, we describe the experiments and discuss results across three diverse geographies g1, g2, and g3 to ensure our approach is generic and makes a positive impact across geographies with different address standards, writing styles, and language variations. these geographies belong to the south america, europe, and asia continents. our experiments leverage historic delivery information and address text that contains information related to building, street, landmark, postal code, etc. our proposed approach holds fair for all types of addresses, for example, urban, rural, commercial, household, etc., and locations. the structure of addresses and writing styles are diverse for these geographies; hence, the improvements observed across all these geographies confirm the wide applicability and generic nature of our approach. while we have limited the evaluation to certain geographies in this paper, our approach is robust for all types of addresses across any geographical continent. the positive results observed across multiple pairwise-matching and geocoding metrics demonstrate the efficacy and effectiveness of our approach. ## real-world application address matching is a fundamental problem to many business applications. in this section, we highlight the positive impact of improving address matching for geocoding and highlight the impact observed via online a/b experiment. ## analysis we analysed our approach and show that it offers the best quality, latency, and operating cost. ## conclusion we proposed a novel framework for pretraining and fine-tuning lms aimed at address matching. it integrates concepts from graph theory and weak supervision with address text and geospatial semantics to generate informative and diverse pairs, thus facilitating pretraining and fine-tuning in a self-supervised manner. we introduced neighbour relation prediction (nrp) as a new pretraining objective. we deployed our approach for real-time geocode learning and presented results from online a/b experiments. we observed improvement in delivery precision and reduction of delivery defects. this led to better delivery planning, decrease in operation costs, and better customer experience.