H e L ju@ V ar D ial 2020: Social Media Variety Geolocation with BERT Models (2020)

Abstract: This paper describes the Helsinki-Ljubljana contribution to the VarDial shared task on social media variety geolocation. Our solutions are based on the BERT Transformer models, the constrained versions of our models reaching 1st place in two subtasks and 3rd place in one subtask, while our unconstrained models outperform all the constrained systems by a large margin. We show in our analyses that Transformer-based models outperform traditional models by far, and that improvements obtained by pre-training models on large quantities of (mostly standard) text are significant, but not drastic, with single-language models also outperforming multilingual models. Our manual analysis shows that two types of signals are the most crucial for a (mis)prediction: named entities and dialectal features, both of which are handled well by our models.

## introduction until 2019, all vardial evaluation campaigns have focused on classification tasks where the number of linguistic varieties was defined beforehand by the task organizers. the 2020 smg task breaks with this tradition and asks the participants to predict latitude-longitude coordinates, i.e., two output values on a continuous scale. the smg task is divided in three subtasks focusing on different linguistic areas: the bosnian-croatian-montenegrin-serbian (bcms) language area, the german-speaking area of germany and austria (deat), and german-speaking switzerland (ch). all three datasets are based on social media data, twitter in the case of bcms and jodel in the case of deat and ch. this paper describes the helju (helsinki-ljubljana) submission to the smg task. our motivation was to investigate how existing classification and regression approaches can be adapted to a double regression task. most of our work is based on the bert sentence classification architecture in both constrained and unconstrained settings. we experiment with various pre-trained models, different types of coordinate encoding and other hyperparameters. finally, we manually analyze the development set predictions made with our best-performing models. ## related work one of the first works focusing on predicting geolocation from social media text is han et al. (2012). the authors investigate feature (token) selection methods for location prediction, showing that traditional predictive algorithms yield significantly better results if feature selection is performed. there has been already a shared task on geolocation prediction at wnut 2016 (han et al., 2016). the task focused not only on predicting geolocation from text, but also from various user metadata. the best performing systems were combining the available information via feedforward networks or ensembles. thomas and hennig (2018) report significant improvements over the winner of the wnut-16 shared task by learning separately text and metadata embeddings via different neural network architectures (lstm, feedforward), merging those embeddings and performing the final classification via a softmax layer. to the best of our knowledge, large pre-trained language models such as bert have not yet been applied to the problem of geolocation prediction, but just to language identification (bernier-colborne et al., 2019). ## data the vardial evaluation campaign provides training, development and test data for the three subtasks. the task organizers also provide a simple baseline for the geolocation task. they compute the centroid ("average location") of all instances in the training data and then predict the coordinates of this centroid for all development or test instances (see ## experiments the geolocation task takes text as input and produces two real-valued outputs, the predicted latitude and longitude. the two outputs can be produced by independently trained models or by a single model that benefits from some form of parameter sharing. ## error analysis figure 1 shows the spatial distribution of the development set instances of the three tasks. the plots on the left show the predicted locations, with the color indicating the distance to the true locations. in all three plots, high-accuracy zones can be distinguished (colored in red), which correspond to urban areas. furthermore, instances that are hard to classify are put into a "default" zone close to the centroid location, in order to minimize distances. the plots on the right show the true locations, with the color indicating the distance to the predicted locations. for the deat task, red and blue dots occur simultaneously in the same areas. for bcms, the major urban areas can again be distinguished easily. we manually analyzed a subset of development set instances for each task. to this end, we chose two areas per task that can be delimited easily and that lie outside of the "default area". we then selected at most 50 "good" and 50 "bad" predictions, on the basis of a task-specific threshold set according to a natural break occurring in the data. we then counted how many of these instances contain dialectal features and named entities (mostly place names, but also names of well known locations, people, institutions or businesses), and whether these features were used consistently (e.g. an instance located in berlin mentions berlin) or inconsistently (e.g. an instance located in berlin mentions köln). this small-scale analysis shows some striking differences between the tasks. for the ch task, good prediction performance is mainly associated with the presence of consistent dialectal features, whereas for the deat task, prediction almost exclusively relies on the presence of consistent named entities. the bcms task lies between those two, with both named entity and dialectal signal being present in good task region consistent features inconsistent features bcms zagreb kolindin ili josipovicev fejk osmijeh cao cao #svenasvetu da uvik slikaju iste pičine belgrade letnji pljusak je okej volio bih da ti kazem ustala sam posle 3 meseca pre 4 sata upravo sam saznao ovu pretužnu vijest ch wallis und wier chännes miterläbu! hahaha je confirme y a trop . . . ich sägu minum botsch ästagsch öi h. hans liebte franz doch franz . . . sust nu jemand moru en statisterolla? da isch en aargaueri[n] uf de . . . chur leid tuats miar glich abitz bisch demfau z frouefäud? wär muos am 3.7 nach airolo irucka? wulewu kuschee awek mua? liabi lüüt fahrend doch eifach 50 wo konn men do heint fortgian? deat berlin dat isch nüscht vienna spoats eichs bitte ma jt zu sagen -also is bei mir doch nicht so fad eine ur schirche kartoffelnase ## conclusion in this paper we have presented the first use of the popular transformer-based bert systems on geolocation prediction that ensured a very strong first place in the unconstrained version of the vardial 2020 smg shared task and the first place in two out of three subtasks in the constrained version of the shared task. we have shown that pre-trained models perform drastically better than the usual traditional machine learning approaches, even if both are based on the same significant amount of data available. models pre-trained on large amounts of text do outperform those pre-trained on the training data only, but not with a drastic difference. this lack of a larger difference might probably be followed back to the lack of dialectal features in the large pre-training data. language-specific pre-trained models do outperform multilingual models, but not as drastically as one would expect. this lack of a larger difference can probably again be followed back to the lack of dialectal features in the pre-training data. finally, performing various target transformations from the original (latitude, longitude) space does not improve results. our error analysis has shown that the three subtasks are quite different regarding the useful signal available. while the deat decisions mostly rely on named entities, and the ch decisions on dialectal features, the bcms subtask lies somewhere in the middle, with both signals having similar importance. by having such a diverse test bed, we believe that we have shown that the current state-of-the art natural language models produce also the best results on this task, regardless of the type of the useful signal, as long as a significant amount of data (around million instances) is available for pre-training and finetuning.