Social Media Variety Geolocation with geo BERT (2021)

Abstract: This paper describes the Helsinki–Ljubljana contribution to the VarDial 2021 shared task on social media variety geolocation. Following our successful participation at VarDial 2020, we again propose constrained and unconstrained systems based on the BERT architecture. In this paper, we report experiments with different tokenization settings and different pre-trained models, and we contrast our parameter-free regression approach with various classification schemes proposed by other participants at VarDial 2020. Both the code and the best-performing pre-trained models are made freely available.

## introduction the smg (social media geolocation) task was first introduced at vardial 2020 @xcite . in this task, the participants were asked to predict latitude-longitude coordinate pairs. this contrasts with most other vardial tasks, in which the goal is to choose from a finite set of variety labels. the second edition of the smg task is run at vardial 2021 @xcite , with the same three language areas as in the previous year: the bosnian-croatian-montenegrin-serbian (bcms) language area, the german language area comprised of germany and austria (de-at), and german-speaking switzerland (ch). all three datasets are based on social media data, twitter in the case of bcms @xcite and jodel in the case of de-at and ch @xcite . this paper describes the helju (helsinki-ljubljana) submission to the smg task. following our successful participation in 2020 @xcite , we again propose systems based on the bert architecture in both constrained and unconstrained settings. we report experiments with different tokenization parameters and with newly available pre-trained models. furthermore, we compare our parameter-free double regression approach with three classification schemes proposed by other participants at vardial 2020. the models and code used in our final submissions are available at https://github.com/clarinsi/geobert . ## related work one of the first works focusing on predicting geolocation from social media text is @xcite . the authors investigate feature (token) selection methods for location prediction, showing that traditional predictive algorithms yield significantly better results if feature selection is performed. there has been already a shared task on geolocation prediction at wnut 2016 @xcite . the task focused not only on predicting geolocation from text, but also from various user metadata. the best performing systems combined the available information via feedforward networks or ensembles. thomas and hennig (2018) report significant improvements over the winner of the wnut-16 shared task by separately learning text and metadata embeddings via different neural network architectures (lstm, feedforward), merging those embeddings and performing the final classification via a softmax layer. during the last iteration of the vardial social media geolocation shared task @xcite , most teams submitted constrained systems, i.e., they did not make use of any form of transfer learning from other tasks or pre-training selfsupervision. two approaches emerged during the shared task -a direct one, modelling the problem as multi-target regression, and an indirect one, which converts the coordinates into a finite set of dialect areas and uses a classification model to predict one of the areas. interestingly, both approaches obtained similar results in the constrained setting, while only multi-target regression was tested in the unconstrained one. we will inspect last year's submissions in more detail in section 4.2. ## data the vardial evaluation campaign provides training, development and test data for the three subtasks @xcite . while the official metric of the shared task is median distance (in kilometers) between the gold and predicted geolocations, the organizers report back both the median and the mean distance metrics. in the previous iteration of the shared task, the two metrics showed to correlate in most, but not all cases. ## experiments due to lack of time (the two evaluation campaigns were held just a few months apart due to the *acl conference bidding procedure), all our experiments are based on our successful 2020 submissions @xcite : we use the bert architecture @xcite with a fully connected layer on top of the cls token. this fully connected layer implements double regression with a two-dimensional output vector and mean absolute error loss. we provide both constrained submissions, where bert models are pre-trained from scratch using the vardial training data, and unconstrained submissions, where we use pre-trained off-the-shelf bert models. for all our experiments, we rely on the simpletransformers library, which is built on top of the huggingface transformers library @xcite , and use the same hyperparameters as in our 2020 submissions @xcite ). 1 in the following subsections, we discuss some additional experiments carried out for the 2021 edition. ## conclusion in this paper, we presented our bert-based approach to the problem of social media geolocation. this year we experimented primarily with tokenizer vocabulary size, space discretization and hyperparameter tuning. the hypothesis that a smaller tokenization vocabulary might improve generalizability and performance proved wrong. we showed that it actually performs worse and converges slower than if standard vocabulary size of 30k wordpieces is used. converting the geolocation task to a classification task yielded worse results in all except the bcms subtask. this is probably due to high concentration of bcms twitter data in the few large cities in the respective area. hyperparameter tuning did not yield any consistent improvements and simply selecting the optimal epoch number on development data showed to be the best approach to this problem. similarly to last year, unconstrained systems performed better than constrained systems. we share our code as well as the best-performing models via https://github.com/clarinsi/geobert and the huggingface models repository.