Geo-Seq2seq: T witter User Geolocation on Noisy Data through Sequence to Sequence Learning (2023)

Abstract: Location information can support social media analyses by providing geographic context. Some of the most accurate and popular Twitter geolocation systems rely on rule-based methods that examine the user-provided profile location, which fail to handle informal or noisy location names. We propose Geo-Seq2seq, a sequence-to-sequence (seq2seq) model for Twitter user geolocation that rewrites noisy, multilingual user-provided location strings into structured English location names. We train our system on tens of millions of multilingual location string and geotagged-tweet pairs. Compared to leading methods, our model vastly increases coverage (i.e., the number of users we can geolocate) while achieving comparable or superior accuracy. Our error analysis reveals that constrained decoding helps the model produce valid locations according to a location database. Finally, we measure biases across language, country of origin, and time to evaluate fairness, and find that while our model can generalize well to unseen temporal data, performance does vary by language and country.

## introduction the analysis of twitter and other social media data supports research in numerous domains by providing a measure of population beliefs and behaviors. a key aspect of many of these studies is the contextualization of posts based on the users' location. for example, studies of covid-19 social distancing rely on knowing the location of users and how they move over time @xcite , models of disease spread during pandemics utilize updated information on population movements @xcite , and studies of civil unrest and democratic reforms rely on isolating data from specific geographic areas @xcite @xcite . however, figure 1 : geolocation of a user profile location to structured location. for example, geo-seq2seq correctly maps "waffle house" (a us-based restaurant) to the us, a zip code to brazil, and the farsi name for iran to iran. while some data contains user-provided structured location information, most do not. furthermore, it is increasingly difficult to rely exclusively on the available location tweet metadata. locationspecific information has slowly been losing popularity among users, and twitter has followed suit by removing the ability to add precise coordinates altogether @xcite . previous metadata analyses studies have validated this trend and found a decline in user-provided location information, i.e., coordinates (stopped in 2019) and place objects (declining and only available in 2% of tweets) @xcite . therefore, many researchers rely on twitter geolocation systems, which automatically infer the location of a user or a tweet. most approaches to social media user geolocation utilize tweet-or user-level metadata @xcite , tweet content (including hashtags) @xcite @xcite , and social networks @xcite . these systems examine one or more tweets from one user and resolve the tweet or user to a structured location object from a gazetteer, or a geographical dictionary, such as google maps or geonames. 1 researchers can then filter to a location of interest, or contextualize information based on locations. a drawback of this method is the reliance on hand-crafted rules, which do not cover the diverse range of ways in which users specify locations. not all users fill out the location field for its intended use or may put inaccurate locations @xcite , slang location names, or a variety of other location strings that may be identifiable to people but not to rule-based string matching systems. rather than rely on existing string matching approaches, we propose to learn a sequence-tosequence (seq2seq) model that maps noisy, multilingual, user-provided location strings into a structured location object selected from a database. for example, our system learns that windy city corresponds to the location object chicago, il, us and zhongguo refers to china (see @xcite 1 for more examples). we train our system on tens of millions of tweets that contain both user-authored location profile strings and userprovided structured location information. we integrate our seq2seq model with carmen @xcite , a popular twitter geolocation tool to produce a unique location in the geonames location database. 2 we build on mt5 @xcite and experiment with various types of restrictions (constraints) in a denoising transformerbased seq2seq model, including a trie-based constrained decoding @xcite scheme to ensure the output corresponds to a known location. we find that our system achieves better accuracy and greatly expanded coverage compared to existing systems. finally, inspired by @xcite , we evaluate the fairness of our model with respect to performance across languages, country of origin, and time. we make the following contributions: ## twitter geolocation twitter geolocation tools can focus either on user geolocation (i.e., where is this user based) or tweet geolocation (i.e., where was this tweet written). additionally, a system can examine a single tweet or all information about a user. our focus is on ascertaining a user's primary location based on their profile information, which remains constant across all of their tweets but can be extracted from a single tweet. the location of a tweet can be identified through: (1) the coordinates embedded in the metadata of the tweet @xcite , (2) the user-provided place metadata @xcite , and (3) inference from the tweet content @xcite @xcite @xcite . the location of a user can be ascertained through (1) aggregated tweet locations from many geotagged or location-inferred tweets (from previously mentioned methods), (2) the user's location string in their profile @xcite , or (3) social network information @xcite . the carmen @xcite geolocation tool infers a location for a user from a single tweet by looking for place metadata, provided coordinates, and (mostly) using a rulebased parser that maps a profile user location string to an internal location database based on geonames. we will utilize this rule-based parser as a comparative baseline for our method. 3 since our focus is on using the user profile location string alone, we omit comparisons to geolocation systems that use other methods. we choose this approach due to speed, privacy, and the prominence of location profile data. geo-seq2seq is fast because the input is only the user profile location string, as opposed to requiring multiple tweets from a user for content analysis or making numerous twitter api calls to gather a user's friends for a social network analysis. further, this method can work on any pre-collected tweets with user profile information, which is advantageous due to the march 2023 depreciation of the free api tier. 4 regarding privacy, we only use information freely provided by the user. we discuss this further in section 9. finally, as shown by @xcite ; @xcite , profile location strings are the only location-related metadata consistently provided by users through the years (60%), unlike place and precise coordinates, which are provided 2% of the time and have been removed, respectively. for these reasons, the comparison methods outlined above are not relevant. further, while other methods have provided baselines against twitter-world and twitter-us @xcite , these datasets are english-only, so including them in this work would not demonstrate the multilingual ability of our approach. ## data the goal of our system is to map a free text string from a user's profile location field to a known structured place name. the location field contains diverse types of content (see figure 1 ), some of which may map to a specific city, or only a country, or no known place. we will learn these mappings based on a large corpus of historical twitter data. common practice is to treat the place object in the tweet metadata as the ground truth location of a user (see figure 2 ). 5 this metadata is included when a user chooses to add it to their tweet. it contains a formal place name of a city, an administrative region (e,g. state), or a country. 6 however, only 2% of the tweets contain a place object @xcite . the place object is an accurate but scarce source of geolocation information. in comparison, 30% to 40% of users (amounting to 60% of tweets) fill out the twitter profile location string. being a free-text field completed by the user, the profile string contains informal location names, made-up locations, or jokes by the users. the profile string is a noisy but abundant geolocation information source. we frame our task as a supervised learning problem where the goal is to translate the noisy profile location string into the structured place object (or an equivalent representation). by collecting tweets with both, we can create a large supervised dataset for training. ## methods we utilize an encoder-decoder transformer-based model to learn a mapping from user location profile strings to structured place strings. given the multilingual nature of our dataset, we select the multilingual t5 model (mt5) @xcite . as discussed in section 3, we add three special tokens: <city>, <admin>, <country>, and fine-tune the embeddings 7 for these tokens along with the model. we fine-tune mt5-small for our task on the 33.4m training examples with the adam optimizer for cross-entropy loss for 5 epochs. all decoding methods use the same pretrained model unless stated otherwise. training details are in appendix a. we call our model geo-seq2seq. 7 initialized with the default hugging face settings of random weights. ## comparison methods we include several baseline methods for comparison against our proposed model. ## evaluation we evaluate all models from three perspectives: coverage, geolocation accuracy, and the validity rate of generated location strings. coverage we define coverage as the fraction of tweets that were resolved to a location. a tweet is "resolved" if the geolocation system successfully proposed a candidate location given the user location string. the coverage metric is similar to recall, but does not consider whether the prediction is correct. ## experimental results we evaluate the generalization effectiveness of the best version of geo-seq2seq (constrained decoding with beam size of 16; see ablation results in section 7.1) by comparing it to other methods on our geolocation dataset. geo-seq2seq greatly outperforms the rule-and memorization-based models, showing that our model has learned to generalize to unseen locations. with respect to coverage, the rule-based carmen profile resolver performs the worst, followed by the table lookup baseline, only providing locations for 53% and 82%, respectively (see results in the remaining metrics evaluate accuracy with respect to geolocation and structured prediction format, specifically whether the output is in the @xmath0, <admin>, <@xmath1, and whether each slot contains a location in the carmen location database. note that the output from geo-seq2seq and the carmen-augmented model achieve a perfect score of 1.0 because they are forced to output valid locations through constrained decoding. the non-constrained methods, table lookup and ground truth, have similar validity rates due to being based on twitter place names, not all of which are present in carmen's location database. with regards to geolocation accuracy, we look at the match ratio for each country, admin, and city slot. while geo-seq2seq by itself has a very high accuracy of 85% and 68% for country and admin, respectively, the carmen-augmented model has higher city accuracy at 34% (versus 31%). this improvement in granularity at the city level suggests the integrated model is better suited for tasks that require finer demographic granularity. ## performance across demographics metrics over the entire test set can hide biases in model behavior on specific sub-groups. when used as part of an analysis pipeline these biases could change study conclusions. for example, @xcite exclude non-english tweets since location based on language id (e.g. japanese tweets come from japan) may portray an unrealistic picture of model performance. we conduct a language and location analysis to determine the fairness of the best performing geo-seq2seq model as measured on the test set. language bias does the language of the profile location bias model behavior? we define a prediction as "language-biased" if the predicted country's primary language, as identified by geonames, is the same as the language of the source location string. since english is prevalent around the world, we remove countries that have english as a primary language for this experiment, leaving 115 out of 146 countries. 10 the list of countries included in the analysis is in appendix appendix d. our model predicted one of the remaining 115 countries for 538k test set examples, and we identified 244k as "language-biased." among the language-biased predictions, 231k (94%) are "correctly biased," meaning the predicted country correctly matches the target country. thus, only 6% of the predictions are wrongly biased by the language of the profile location. further analysis is needed in languages prevalent in multiple countries. ## conclusion we present geo-seq2seq, an mt5 model finetuned for twitter user geolocation through denoising user profile location strings. we train it on twitter-pug, a dataset of 35.4m location strings with ground truth labels. our model outperforms existing systems with 99% test set coverage and 85% prediction accuracy at the country-level. augmented with carmen the model achieves 34% city accuracy, improving over carmen's 14% accuracy. the success of the model comes from a constrained decoding strategy with a beam size of 16, with a "reversed" target string. additionally, we breakdown performance by location and language, highlighting biases in model behavior. future work should concentrate on producing models that are fairer with regard to locations and languages.