Regressing Location on Text for Probabilistic Geocoding (2021)

Abstract: Text data are an important source of detailed information about social and political events. Automated systems parse large volumes of text data to infer or extract structured information that describes actors, actions, dates, times, and locations. One of these sub-tasks is geocoding: predicting the geographic coordinates associated with events or locations described by a given text. I present an end-to-end probabilistic model for geocoding text data. Additionally, I collect a novel data set for evaluating the performance of geocoding systems. I compare the model-based solution, called ELECTRo-map, to the current state-of-the-art open source system for geocoding texts for event data. Finally, I discuss the benefits of end-to-end model-based geocoding, including principled uncertainty estimation and the ability of these models to leverage contextual information.

## introduction text data are an important source of information about social and political events. we introduce a novel method for predicting the latitude and longitude of locations mentioned or described in natural language texts ("geocoding"). this neural networkbased method offers several advantages over existing rule-based techniques for geocoding: (1) it produces a probability distribution over predicted latitudes and longitudes thereby allowing users to report the certainty of their estimates; (2) it does not require the identification of place names in the text prior to geocoding; (3) it naturally leverages contextual clues to improve predictions and disambiguate location names. this paper proceeds by first providing a brief overview of related work in geocoding and language modeling. we then introduce a probabilistic model for geocoding texts and identify a dataset with which to train and evaluate the model. we compare our results to existing methods and conclude with suggestions for future research. mordecai is an open source tool for steps 1 and 2 (halterman, 2017). mordecai uses a pretrained named entity recognition model and word2vec (mikolov et al., 2013) to match location names identified within an unstructured text document to known locations within the geonames gazetteer (geonames). kulkarni et al. (2020) present a model-based geocoding solution. their convolutional neural network model predicts geographic grid cell membership for each input text; it does not predict latitude and longitude values directly. this complicates comparison with the model presented here which directly regresses latitude and longitude on text. for example, the evaluation metrics the authors chose for their model are largely based on classification accuracy rather than continuous measures of nearness, as would be the case in a regression setting. 1 ## model we introduce a model that is capable of performing steps 1 through 3 ( § 1.1) end-to-end. that is, given training data exemplary of the desired mapping from text inputs, x, to geographic coordinates, y, this model is fine-tuned such that it learns a function f (x i ; w) → ŷi , where w is the set of model parameters. this is a non-linear multivariate regression of latitude and longitude on text. we modify a pretrained distilroberta model by adding three fully-connected dense layers with sigmoid activation, an output ("head") layer, and a custom loss function. we use this model to minimize the negative log likelihood of a five component mixture of von mises-fisher (vmf) distributions conditional on the input text. the von mises distribution is an approximation of a univariate gaussian distribution on the circumference of a circle. the vmf distribution generalizes the von mises distribution beyond two dimensions to the surfaces of spheres and hyperspheres; when @xmath0, the vmf distribution is equivalent to the the von mises distribution. because the vmf distribution has support over the surface of the unit p -1 sphere in p euclidean space, we must transform our geodetic coordinates (latitude and longitude) to cartesian coordinates on this sphere. the formulae to do so, assuming a spherical earth, are given by equations 1-3. the vmf probability density function is given by equation 4 . µ, the mean direction, is a point in p space that falls on the unit p -1 sphere. a point x in p space can be projected onto this sphere by l2 normalization: x/||x||. the concentration parameter, κ, controls the dispersion of the distribution across the surface of the sphere. κ = 0 corresponds to a uniform distribution over the entire sphere while κ = ∞ corresponds to a point mass at µ. i p/2-1 is the modified bessel function of the first kind at order p/2 -1. a probabilistic neural network model with a single vmf component is optimized by minimizing the negative log likelihood given in equation 5 . the outputs of the neural network, given an input text x i , are the parameters of a vmf distribution. therefore, the model estimates a distribution over possible coordinates for a given input text. while the parameters of the neural network itself (w) are deterministic, predicting a probability distribution for each input text allows us to capture aleatoric uncertainty. aleatoric uncertainty is the uncertainty inherent in the data themselves. in the case of geocoding text, this uncertainty may result from texts that do not distinguish between springfield, il and springfield, ga, or from texts that refer to multiple locations (assuming that the model in question is unable to represent a multimodal distribution). this uncertainty is unlikely to be homoskedastic; some texts will more precisely specify relevant locations than others. we allow for heteroskedastic uncertainty by estimating both the central tendency (μ i ) and the dispersion (κ i ) of a target distribution. building on the negative vmf log likelihood loss described above, we optimize a neural network model to predict a mixture of vmf distributions. @xcite for every input text, the model predicts parameters for five vmf distributions in addition to a set of mixing probabilities. the mixing probabilities describe the weights associated with each of the five vmf components. a more flexible distributional shape than it would otherwise be able to with a single vmf component. indexing the mixing components, ρ, by k, the revised loss function is given in equation 6 . we refer to this model as electro-map: end-to-end location estimation with confidence via transformer regression. foot_1 to evaluate electro-map, we collect data from all wikipedia articles with coordinates linked to wikidata.org. foot_2 these data include the primary latitude and longitude associated with an article, globe, title, language, and extract attributes. the data were collected via the official wikipedia api by iterating over the set of wikipedia pages linked to wikidata geographic entries. @xcite together, the data comprise the introductory sections of 1,286,475 english language articles. most of the excerpts are between one sentence and a couple paragraphs in length. many of these texts contain references to multiple geographic locations, but each one only has one "correct" latitude and longitude pair that describes the precise location of the article's referent. these are partitioned into a training set (1,260,746 articles), a validation set (12,864 articles) and a test set (12,865 articles). foot_4 ## evaluation we compare the performance of electromap against mordecai. because mordecai and electro-map can both return multiple results per text, we offer three solutions for aggregating results to a single latitude and longitude prediction per observation. the first is to take the single highest probability prediction (highprob). @xcite the second is to take the best prediction from the mixture (best). @xcite the third is to take a random prediction from the mixture (random). mordecai occasionally returns null results. in these cases, we impute a latitude and longitude pair of (0.0, 0.0). we also provide results for a complete cases analysis of mordecai, omitting all 279 observations for which mordecai failed to produce a geolocation. results are shown in ## conclusion when humans perform geocoding manually, they often rely on contextual clues for assistance. those clues may or may not come from the text itself. for instance, the presence of other named entities, like sports teams, may help human coders to distinguish between washington state and washington d.c. automated processes for geocoding should also make use of contextual clues. model-based geocoding offers a natural method for both incorporating contextual clues and for dealing with the uncertainties that arise while geocoding. electro-map, for instance, quantifies uncertainty by estimating a mixture of probability distributions over likely geographic coordinates. furthermore, model-based geocoding offers the ability to fine-tune for specific tasks: researchers may be interested in geocoding certain parts of texts and not others (e.g. birth and death places). to the extent that the model is unable to distinguish between multiple location types in the source text, this ambiguity should be reflected in the model's reported uncertainty. model-based and gazetteerbased methods (like mordecai) are not exclusive, though. it may be possible to derive better results by, for example, first identifying a distribution over likely locations via a statistical model and then "snapping to" a most likely location within that distribution using a gazetteer. finally, the success of multilingual transformers suggests that electro-map or related techniques may generalize across languages (k et al., 2020). future efforts on model-based geocoding should seek to evaluate cross-lingual performance and measure the importance of context on location disambiguation.