Where on Earth Do Users Say They Are?: Geo-Entity Linking for Noisy Multilingual User Input (2024)

Abstract: Geo-entity linking is the task of linking a location mention to the real-world geographic location. In this we explore the challenging task of geo-entity linking for noisy, multilingual social media data. There are few open-source multilingual geo-entity linking tools available and existing ones are often rule-based, which break easily in social media settings, or LLM-based, which are too expensive for large-scale datasets. We present a method which represents real-world locations as averaged embeddings from labeled user-input location names and allows for selective prediction via an interpretable confidence score. We show that our approach improves geo-entity linking on a global and multilingual social media dataset, and discuss progress and problems with evaluating at different geographic granularities.

## introduction the real-world geographic location of social media users is valuable data for many computational social science tasks, including disaster response @xcite , disease surveillance @xcite , analyzing language variation @xcite , and comparing regional attitudes @xcite . many studies have used twitter (now known as x) data for such analyses, focusing on geo-tagged tweets where each tweet is associated with latitude and longitude coordinates. however, geo-tagging with coordinates was deprecated in june 2019 and even before then only a small percentage of tweets (< 2%) was geotagged @xcite . it has thus become increasingly necessary to infer location from user profiles and especially from the free text location field, in which a user may enter anything they want to identify their location. this field is frequently specified, with at least 40% of users providing recognizable locations in over 60 different languages @xcite . the task of linking a location reference to the actual geographic location is known as geo-entity linking (see in this paper, we investigate the task of geoentity linking for noisy, multilingual user-input location references. our work makes the following contributions: geo-entity linking, also known as toponym resolution, seeks to link some mention of a geographic entity to the correct entity in a target database. previous approaches typically use some combination of the mention's text and/or context; knowledge bases (e.g. gazetteer, wikipedia) which contain features such as population, location type, etc.; and coordinates/geometric features. they may use some mix of rule-based, unsupervised, and supervised methods. the majority of prior work on geo-entity linking has focused on data in the english language and in the domain of news articles @xcite @xcite @xcite @xcite @xcite . we note that the geospatially grounded model geolm -which was trained only on english data -was evaluated on geo-entity linking in a way similar to our proposed method, by ranking locations by cosine similarity between each candidate and the query location @xcite . this is similar to our proposed namegeo method, although the authors did not explore using cosine similarity thresholds for selective prediction or any of the other variations that we investigate. some prior work has examined geo-entity linking in historical texts @xcite @xcite , which includes english, spanish, dutch, and german data; and in web pages @xcite , which includes french, spanish, and italian data. most relevant to the current work are previous studies which have examined geo-entity linking in social media data. alex et al. (2016) entity-linking is the broader task of linking some mention of an entity -which could be a person, place, or organization -to the correct entity in a target database. we note a similarity between our proposed usergeo method and one introduced for entity-linking in @xcite , in that they both represent entities using all mentions in the training data. however, our method is simpler as it does not involve model training. addition-ally, usergeo represents an entity by averaging mention embeddings instead of having a separate embedding for each mention, which induces a more holistic entity representation that can better handle noisy mentions present in social media data. predicting user location, also known as user geolocation, is a task distinct from geo-entity linking social media data in that it seeks to determine the location of a user using both text data and user metadata, including post content, user bio, user language, time zone, or social networks @xcite @xcite @xcite @xcite . the location field is one of many features used for location prediction, if used at all. geoparsing is the task of both identifying and linking geographic entities in unstructured text @xcite , essentially combining toponym recognition @xcite and geo-entity linking. geoparsers focused on standard english texts (including news articles, wikipedia, or scientific papers) include cliff-clavin (d' @xcite , camcoder @xcite , and dm_nlp @xcite . other geoparsers include the edinburgh geoparser for historical english text @xcite , geotxt for english social media data @xcite , and perdido for french texts @xcite . ## accuracy upper bounds to approximate an upper bound for accuracy scores on our dataset, we conducted an analysis of a random sample of user inputs to determine how many contained references to actual locations. the percentage of examples which contain a location at each granularity suggest an upper bound for accuracy, indicating the approximate proportion that a geo-entity linker could be expected to connect to a real geographic location. the first author manually annotated 120 random examples from the training set for whether or not the user input contained an identifiable reference to a location at the country, administrative, and/or city levels. 7 we include results from this annotation in we discuss a few problems with evaluating geoentity linking at the city level for social media data. first, the assumption of the geocoordinates as ground truth is frequently untrue and especially so at the fine-grained city level. unlike news articles or wikipedia data, the place that a user puts on social media is not necessarily the place where they actually are. as @xcite have shown, only 70% of users have geocoordinates within 100km of the place specified in their location field. second, since we define ground truth as the closest city to the geocoordinates with a population higher than 15k, there will always be a mismatch between ground truth and location field if a user puts a city with a lower population. third, there is a disconnect between what is considered a city by a database such as geonames versus by an everyday person. istanbul, jakarta, moscow, gaborone, lampang, and santa ana are all names of both cities and administrative regions in geonames -when users put them in their location field, it's unclear 7 we did not count coordinates as containing a location, as we wanted to investigate the ability of the model to identify natural language location references. if a location was named that could be either an administrative region or a city, then the example was marked as containing both. which one they are referring to. we hypothesize that these problems contribute to the poor performance of geo-entity linking tools at the city level. we additionally acknowledge how inferring location at the city level is more invasive than inferring at the country or administrative level because it predicts more fine-grained information about the user. therefore, due to mismatches between geocoordinates and user-defined location fields as well as the privacy concerns, we suggest that researchers only predict city-level location if it's necessary for a given application and otherwise use country-or adminstrative-level predictions. ## conclusion and future work in this paper we introduced new methods for geoentity linking noisy, multilingual social media data with selective prediction. of our two best performing methods, one does not require any training data (namegeo+variants) while the other achieves state-of-the-art performance at the country and administrative levels (usergeo). we also discussed problems with geo-entity linking at the city level for social media data, and suggested against doing this unless necessary for the application. this work is not without limitations. we do not compare performance with llm prompting methods, as they are too expensive for largescale datasets and it's currently unclear how much prompt testing is necessary for a robust evaluation. however, it would be a useful comparison to know how llms perform at zero-or few-shot geo-entity linking; we leave this for future work. we also acknowledge that our methods rely heavily on sbert models, and thus are reliant on a third party for sustaining them. we plan to release a version of the usergeo location embeddings in future, although we will not release the current version due to concerns about the twitter-global data (e.g. unbalanced across countries, unclear sampling methodology). we also hope to further evaluate our method on other domains with noisy location references, such as historical data, and to explore extensions of our method so it may be used for the broader task of geoparsing unstructured text.