Changes in Tweet Geolocation over Time: A Study with Carmen 2.0 (2022)

Abstract: Researchers across disciplines use Twitter geolocation tools to filter data for desired locations. These tools have largely been trained and tested on English tweets, often originating in the United States from almost a decade ago. Despite the importance of these tools for data curation, the impact of tweet language, country of origin, and creation date on tool performance remains largely unknown. We explore these issues with Carmen, a popular tool for Twitter geolocation. To support this study we introduce Carmen 2.0, a major update which includes the incorporation of GeoNames, a gazetteer that provides much broader coverage of locations. We evaluate using two new Twitter datasets, one for multilingual, multiyear geolocation evaluation, and another for usage trends over time. We found that language, country origin, and time does impact geolocation tool performance.

## introduction demographic studies leverage location-specific social media posts to track impactful events such as civil unrest @xcite @xcite , natural disasters @xcite , and disease spread @xcite . for social media posts from twitter, researchers either collect posts from locations of interest in real-time with the twitter api, or use third-party twitter geolocation tools to identify tweet locations on an existing dataset. in @xcite , the authors distinguish between user and tweet geolocation. we focus on tweet geolocation in this work. these tools identify the location of a user or tweet based on tweet metadata @xcite , tweet content @xcite @xcite @xcite , and social networks @xcite . while widely used, geolocation tools tend to be english-centric and are often not evaluated for global coverage or performance across time and language. these factors are important to study, since available user metadata, twitter policies, and content patterns on which the tools depend on can change significantly over time. in this work, we assess how the following factors impact geolocation tools: we measure performance in geolocation by coverage, i.e. the number of tweets that can be mapped to a location, and accuracy, the correctness of the assigned locations. when evaluated together, these metrics provide analogues to recall and precision, respectively. to answer the above research questions, we analyze the performance of twitter geolocation tool carmen @xcite , across time, language, and country of origin. in order to study performance across these factors, we introduce twitter-global, a new geocoded multilingual and multiyear dataset (2013-2021) of 15.3m tweets. we created this dataset to fill a gap in other geolocation evaluation datasets that are either english-only @xcite , or multilingual but restricted to short periods of time @xcite . we focus on carmen since it is a rule-based tool that can be run quickly on large collections of tweets. since carmen was built for english tweet datasets from 2013, we update the tool and introduce carmen 2.0. this updated version relies on geonames, foot_0 an open-source geographical dictionary, or gazetteer. in contrast to carmen's us and english-centric database, geonames provides global coverage in many languages. through comparisons of geonames-augmented carmen 2.0 with the original carmen location database, we can study the effects of incorporating more non-us and non-english locations on geolocation performance. in addition to studying carmen 2.0's performance with regard to different factors, we also include a longitudinal study of twitter demographics over time from 2013-2021, with respect popularity across different countries, languages, and geolocation metadata. this study is on a collection of 5.7m tweets sampled from the 1% twitter stream, which we refer to as twitter-random. the demographic and metadata analysis provides statistics to support design decisions for researchers developing their own geolocation tools. we contribute the following: all experiment code and data (tweet ids) are released on in the github code repository. ## related work most work in twitter geolocation focuses solely on tool development and performance, usually on english-centric datasets published years ago. in this paper we question how those tools would perform on twitter datasets today, but focus on a single tool, carmen. @xcite analyze the impact of twitter policy changes on research. the authors study tweet metadata availability over time, such as exact coordinate availability and granularity of place objects. most importantly, the authors discuss the impact of the 2019 twitter policy change to remove precise locations from tweets (starting from 2019), and how that affects geolocation tools and researchers who depend on the coordinates. their work limits their study to tweets from 2020-2021, and in our work we study these metadata patterns over a larger span of time, 2013-2021, in addition to the impact of other factors, such as language and country of origin, on geolocation tool performance. we compare our multi-year trend analysis to theirs in section 6. this multiyear analysis is useful for researchers geolocating tweets in older twitter datasets. ## carmen 2.0 in this paper we present carmen 2.0, an updated version of geolocation tool carmen. we aim to increase the coverage and robustness of carmen to language and countries by using a open-source database, geonames. in addition to a new location database, we include other performance improvements, such as compatibility with twitter api v2 (see appendix §b). since carmen 2.0 does not change the core functionality, we focus on the construction and use of the internal location database, and we direct the reader to appendix §a for a review of the location resolvers or @xcite for more details. ## geolocation evaluation through comparing carmen's original database (see §3.1.1) with the new geonames based database (see §3.2), we can answer our research questions from section 1 to see how geolocation tool coverage and accuracy change with respect to language, country of origin, and time. the performance is evaluated with similar metrics of other geolocation tools mentioned in §2. in addition to updating carmen, we also created two datasets to support our analysis. ## experiments as enumerated in §1, we are interested in how the following factors impact geolocation tool performance: language, country, and time. to answer these questions we perform an ablation study over carmen location databases (see §3.2) and different subsets of twitter-global (see §4.1). ## longitudinal analysis of twitter user location we have seen how using a less biased geolocation tool offers better performance with respect to coverage and accuracy. however, despite the overall better performance with the geonames-combined location database, coverage still varied greatly when evaluated over time, as in section 5.3. to better understand this performance difference and to provide insights for other geolocation researchers, we present a longitudinal study of trends in location metadata availability and user demographics. all metadata and demographic statistics are gathered from twitter-random (see §4.1). ## ethical concerns two issues that arise when geolocating users on social media: (1) privacy concerns and (2) consequences of incorrect predictions. the privacy concerns are related to surveillance and revealing sensitive locations of users, such as their home address. since carmen only uses metadata provided by the user in the form of tagged places, coordinates, and profile location, it only infers locations readily shared by users. @xcite provide a helpful discussion of applications that require differing levels of location granularity, such as disaster relief or disease spread requiring more precise information (high granularity) versus marketing campaigns or opinion tracking (low granularity). an issue with low granularity arises in high-risk applications where low precision is not helpful, such as tracking disease spread within a country. a possible solution for balancing higher granularity and user privacy is to map a user's location to the largest city closest to the user. carmen 2.0 does this automatically since the database only contains cities with population greater than 15000. there can be negative consequences to using incorrectly inferred locations, such as in tracking high-risk emergencies like disease spread and civil unrest. geolocation tool performance ablation over granularity, language, and country, is important for researchers to make informed decisions about location accuracy. ## conclusion geolocation tweets is useful for researchers that need to filter tweets to those originating in specific locations to study health, opinions, etc, by demographic. in this work we study and discuss the impact the factors of language, country origin, and time, can have on tweet geolocation. to support our study we introduced carmen 2.0, an updated version of geolocation tool carmen @xcite backed by an open-source database, geonames. in addition to the tool, we introduced two datasets: (1) twitter-global, a twitter geolocation evaluation dataset for language, country, and time ablation studies, and (2) twitter-random, a sample of the worldwide twitter stream from 2013-2021 for studying general country and language demographics and metadata availability over time. we found a significant difference in performance in the ablation, with higher performance for english and us-based tweets. also, we provided trends in metadata availability from 2013 to 2021, and discuss reasons for the decline in coordinates and place metadata. for future work in twitter tweet geolocation, we encourage the use of content and metadata fields, such as user profile location. focus on these consistently available metadata can make a tool robust to policy changes. also, we encourage evaluating the geolocation model across language, time, and countries to ensure fair performance.