Geolocation of Tweets with a B i LSTM Regression Model (2020)

Abstract: Identifying a user’s location can be useful for recommendation systems, demographic analyses, and disaster outbreak monitoring. Although Twitter allows users to voluntarily reveal their location, such information isn’t universally available. Analyzing a tweet can provide a general estimation of a tweet location while giving insight into the dialect of the user and other linguistic markers. Such linguistic attributes can be used to provide a regional approximation of tweet origins. In this paper, we present a neural regression model that can identify the linguistic intricacies of a tweet to predict the location of the user. The final model identifies the dialect embedded in the tweet and predicts the location of the tweet.

## introduction social media platforms are useful sources for doing data analysis on linguistic patterns. twitter is one such platform and allows access to tweets posted by the users via apis. while the metadata of a tweet is accessible, only 3% of tweets include geotagging information @xcite . this severely limits the use of a metadata-based approach for location identification. even if a geolocation model performs well with such metadata, it will fail in the absence of such information. since tweets are rich in information, a model that relies only on the textual content of tweets will have a better chance of performing consistently. the location prediction task can be addressed either as a classification problem or a regression problem. the former can be implemented in two ways; by treating known locations as classes or by dividing the entire region at hand into a grid and using the center of the sub-regions as classes. a regression approach can be seen as a simple double regression task where a model predicts the latitude and longitude of a tweet separately, based on its features. previous studies have shown that tweets, along with their metadata, can be used to predict a user's location @xcite . neural models have been used to better utilize the metadata, along with the tweets @xcite . this paper presents an approach to predict the location of a user using a neural model trained solely on the tweets' text content, without any external knowledge sources. this allows the model to generalize more easily to new domains and languages. this paper intends to get justifiable results at a low cost and with limited resources. the remainder of this paper is organized as follows: the first section provides an overview of related works on location prediction. section 3 described the datasets used. section 4 describes the details of the neural network architecture. results on the validation set are shown in section 5. finally, section 6 concludes the paper with possible future work. ## related work one of the earliest works on text-based geotagging @xcite used named entities such as cities, states, etc. mentioned in the text to identify the geographic scope of the web pages in a classification setup. later, rule-based methods were employed to identify the locations @xcite , while others used named entity recognition in combination with various machine learning techniques @xcite . all these approaches rely on the fact that there is a mention of some location in the text which may not always be the case. other approaches include supervised classification @xcite using words as features and unsupervised learning, where clustering techniques are employed based on topic modeling @xcite . some of the previous research has focused on leveraging the metadata associated with the tweet to improve the performance. many results submitted in the workshop on noisy user-generated text (wnut2016) @xcite used such metadata heavily. models were built that focused on userdeclared locations, timezone values, and user self-descriptions in addition to the content of the tweet itself @xcite . as part of feature pre-processing, tweet-level geo coordinates were repalced with geo coordinates of the corresponding tweet cities using several mapping services such as geonames and time zone boundaries. finally, neural networks were trained using the fasttext n-gram model @xcite on posted text, user location, user descriptions, and user timezones. ensemble methods are also common, often relying on several information resources like text, user location text, user time zone information, messenger source, and reverse country look-ups for url mentions. @xcite relies on specific url mentions and screened website metadata for geographic coordinates. multinomial naïve bayes methods have been employed that focused on the use of textual features (i.e., location indicative words, geonames gazetteers, user mentions, and hashtags) @xcite . since the authors are using location indicative words, city/country names, hashtags and @mentions as a combined feature set, even though it relies on textual data more than the metadata, dialect related information isn't covered. related work to date covers a wide range of languages and dialects including dutch @xcite , british @xcite , american english @xcite and african american vernacular english @xcite . ## data set we use the data provided for smg sub-task @xcite of the vardial 2020 shared task is used for the training and testing purpose of the model @xcite . using data from social media platforms like twitter and jodel, three language areas are covered. each language has a training, a validation, and a test file. each file contains entries separated by newlines and each entry contains a latitude, a longitude, and a text tuple, separated by tabs. the distribution of all three datasets across latitudes and longitudes are shown in figure 1 , figure 2 , and figure 3 below. figure 2: dataset distribution of ch jodels across latitudes and longitudes figure 3: dataset distribution of bcms tweets across latitudes and longitudes ## architecture to identify the location of a given tweet, a double regression approach has been taken. the model will identify the latitude and longitude separately for a given text and the distance between the actual and predicted location will be used to measure its performance. for this purpose, we are using two bidirectional lstm models. for text pre-processing, we use a tweet tokenizer as part of the cleanup, to remove unwanted punctuation and lower-case conversion. we use a list of german stopwords to further clean the de-at dataset. we decided to keep the stopwords for swiss german due to the smaller size of the dataset. for bcms, stopwords from multiple languages are needed which are not as consistent as german, so those are included in the dataset as well. using tf-idf weighting allows us to focus on the highly relevant tokens in the text. using this new set of relevant tokens, the tweets are updated so that irrelevant tokens are removed. a fasttext model is trained on this new, modified training set, unsupervised. this trained fasttext model is used to produce an embedding matrix. the unsupervised training allows the overall model to be implemented in a similar fashion for all 3 languages. a neural model is created with a trainable embedding layer, an embedding matrix from the fasttext model is used for obtaining embeddings, and two bidirectional lstm layers are used, one for latitude training and one for longitude. using a trainable embedding layer with the embedding matrix reduces the training epochs required for the model to converge. since both lstm layers share a common embedding layer, both have to rely on the same set of embeddings for learning. two dense layers are added before the output layer. this allows the model to adjust the weights better for a gradual change. a visual representation of the architecture is presented in fig. 4 . because the evaluation metric for this task is the median distance loss over the test data, quantile function with 0.5 as alpha is used as the loss function. while a neural model with mse or mae as the loss function predicts better for individual tweets, the distance loss over the entire set becomes greater than the quantile function. on the other hand quantile function takes into account the distribution of the dataset while training and, with alpha set to 0.5, it estimates the median instead of mean. for regularization, since l1 brings weights of unimportant features to zero, only contributing features remain. this helps in better feature selection. we use relu as activation functions and adagrad as the optimizer for adaptive updates of frequently and infrequently occurring feature parameters. ## results the performance of the model on validation and test data is shown in it can be observed that the deviation of validation and test result is not more than 10km for de-at and bcms dataset and much smaller for the ch dataset. the marginal deviation on the ch dataset can be attributed to the small size of the dataset. another factor responsible for the scale of deviation for de-at and bcms vs ch is the range of latitudes and longitudes. below is the range of latitudes and longitudes for all 3 datasets. from ## conclusion and future work we have presented a neural network architecture that addresses the tweet location prediction as a dual regression task. no custom processing of the text was done besides a generic tweet tokenization. as part of the cleanup, stopwords were removed from the standard german dataset only. no pre-trained model or embedding was used which made the model generic enough to be used for any dataset. the model was trained on google colab with limited gpu access with a maximum training time was around 8hrs. in the future, we aim to use the distance between the predicted and actual locations as the main loss parameter to train the model. this should give a better insight into the usage of quantile function since this task focused on median distance loss as the performance metric. we will also analyze the performance of our model against a classification model where the known tweet-emission locations are used as a constrained set of labels.