U nibuc K ernel: Geolocating S wiss G erman Jodels Using Ensemble Learning (2021)

Abstract: In this work, we describe our approach addressing the Social Media Variety Geolocation task featured in the 2021 VarDial Evaluation Campaign. We focus on the second subtask, which is based on a data set formed of approximately 30 thousand Swiss German Jodels. The dialect identification task is about accurately predicting the latitude and longitude of test samples. We frame the task as a double regression problem, employing an XGBoost meta-learner with the combined power of a variety of machine learning approaches to predict both latitude and longitude. The models included in our ensemble range from simple regression techniques, such as Support Vector Regression, to deep neural models, such as a hybrid neural network and a neural transformer. To minimize the prediction error, we approach the problem from a few different perspectives and consider various types of features, from low-level character n-grams to high-level BERT embeddings. The XGBoost ensemble resulted from combining the power of the aforementioned methods achieves a median distance of 23.6 km on the test data, which places us on the third place in the ranking, at a difference of 6.05 km and 2.9 km from the submissions on the first and second places, respectively.

## introduction the social media variety geolocation (smg) task was proposed, for the second year consecutively, in the 2021 edition of the vardial evaluation campaign @xcite . this task is aimed at geolocation prediction based on short text messages exchanged by the users of social media platforms such as twitter or jodel. the location from where a short text was posted on a certain social media platform is expressed by two components: the latitude and the longitude. naturally, the geolocation task is formulated as a double regression problem. twitter and jodel are the platforms used for data collection, and similar to the previous spin of smg at vardial 2020 @xcite , the task is divided into three subtasks, by language area, namely: the focus of our work falls only on the second subtask, smg-ch, tackled via a variety of handcrafted and deep learning models. we propose a single ensemble model joining the power of several individual models through meta-learning based on extreme gradient boosting (xgboost) @xcite . we trained two independent ensemble models, each predicting one of the components that form the geographical coordinates (latitude and longitude). the first model plugged into our meta-learner is a support vector regression (svr) model @xcite based on string kernels. previous usage in dialect identification has proved the efficiency of this technique in the task of interest @xcite @xcite . the second model included in the ensemble is a hybrid convolutional neural network (cnn) @xcite that combines, in the same architecture, character-level @xcite and word-level representations. the ability of capturing morphological relationships at the character level and using them as features for cnns is also known to give promising results in dialect identification @xcite . different from works using solely character-level cnns for dialect identification @xcite , we believe that the addition of words might bring the benefit of learning dialectspecific multi-word expressions that are hard to capture at the character level @xcite . bidirectional encoder representations from transformers (bert) @xcite is a top performing technique used in recent years for solving mainstream nlp problems. thus, it seems fit to also include the outputs of a fine-tuned german version of bert in our xgboost meta-learner. we conducted experiments on the development set provided by the shared task organizers @xcite in order to decide which model to choose as our submission for the smg-ch subtask. our results indicate that the ensemble model attains the best performance. with median distances that are 5-6 km higher, all the other models, tested individually on the development set, provide slightly worse predictions. the remainder of this paper is organized as follows. we present related work on dialect identification and geolocation of short texts in section 2. our approach is described in detail in section 3. we present the experiments and empirical results in section 4. finally, our conclusions are drawn in section 5. ## related work our study of the related work starts with a brief overview of geotagging based on text. then, we look at more specific methods studying geotagging in social media and we also investigate the amount of research done, from a computational linguistics perspective, in geotagging by dialect, with focus on last year's approaches for the same subtask, namely smg-ch at vardial 2020. text-based geotagging. the works based on text to perform geotagging can be divided into three generic categories by the approaches taken in order to predict location. the first type of approach relies mainly on gazetteers as the source of the location mappings. this tool is adopted in a number of works @xcite @xcite , from ruled-based methods @xcite to various machine learning techniques that rely on named entity recognition @xcite @xcite . this category of methods brings the disadvantage of relying on specific mentions of locations in text, rather than inferring them in a less straightforward manner. these direct mentions of places are not a safe assumption, especially when it comes to social media platforms which represent the data source in some of these studies @xcite . the other two main categories of approaches for text-based geolocation rely on either supervised @xcite or unsupervised @xcite @xcite learning. the latter methods usually employ clustering techniques based on topic models. geolocation in social media. a number of works @xcite look at this task from a supervised learning perspective. however, in these studies, other details (e.g. social ties) in the users profile are considered rather than their written content. other works in this area are related to our current interest in studying language variation for the geolocation of social media posts @xcite @xcite @xcite . among these, various machine learning techniques are employed in location prediction, ranging from probabilistic graphical models @xcite and adaptive grid search @xcite to bayesian methods @xcite and neural networks @xcite . dialect-based geolocation. many dialects are covered in the text-based geotagging research to date, including dutch @xcite , british @xcite , american @xcite and african american vernacular english @xcite . out of all the languages that were subject to location detection by dialect, we are interested in german. in this direction, the study that is the most relevant to our work is that of @xcite which targets the german language and its variations. approximately 16.8 million online posts from the german-speaking area of europe are employed in this study with the aim of learning document representations of cities. a small fraction of these posts are jodels collected from the german speaking side of switzerland, and these are also used in the smg-ch subtask that we are addressing in this paper. the assumption here is that the methods should manage to capture enough regional variations in the written language, which can serve as the means to automatically distinguish the geographical region of social media posts. the verification performed in this direction in the original paper @xcite used clustering to determine larger regions covering a given dialect. however, given the shared task formulation, we take a different approach and use the provided data in a double regression setup, addressing the problem both from a shallow and a deep learning perspective. as previously mentioned, this is the second consecutive year in which smg-ch is featured at @xcite , with a similar format, although an updated data set. the participants of the 2020 smg-ch shared task @xcite studied this task from a variety of angles. some techniques are based on deep neural networks such as the popular bert architecture @xcite , bidirectional long short-term memory (lstm) networks applied on fasttext embeddings @xcite or character-level cnns @xcite . other techniques are based on shallow or handcrafted features such as a gridbased prediction using an n-gram language model @xcite or a clustering technique that shifts the problem into a discrete space, then uses an svm for the classification of posts into regions. our best submission @xcite in last year's campaign was an ensemble based on xgboost as meta-learner over the predictions of three different models: ν-svr with string kernels, a character-level cnn and an lstm based on bert embeddings. this year, we acknowledge that our deep learning models had sub-optimal results in our previous participation at smg-ch. consequently, for this year, we bring stronger neural networks into the xgboost ensemble. instead of using an lstm with bert embeddings, we finetune the cased version of german bert and use it directly for double regression in a setup that is more suitable to the data set size, compared to our previous endeavour. instead of a character-level cnn, we use a hybrid cnn which learns end-to-end rep-resentations for both words and characters. the aforementioned deep learning models are plugged into an xgboost ensemble alongside a retrained version of the ν-svr model employed in our past participation @xcite . our current changes introduce a significant improvement in median distance compared to our previous results, on a similar (and perhaps more challenging) data set. ## method the only system that our team submitted for the smg-ch subtask is an ensemble model based on the xgboost meta-learner, as illustrated in figure 1. in this section, we describe the three machine learning techniques that provide their predictions as input for the meta-learner, as well as the gradient boosting method that combines the independent models. ## conclusion in this paper, we proposed an ensemble learning model for the geolocation of swiss german social media posts. the ensemble is based on an xgboost meta-learner applied on top of three individual models: a hybrid cnn, an approach based on string kernels and a fine-tuned german bert model. given the final results obtained in the smg-ch subtask, we conclude that predicting the location of swiss german social media posts is a challenging task, the median distance being higher than 20 km. using external data sources to build a language model seems to be a more promising path towards success, as shown by the final standings of the vardial 2020 @xcite @xcite smg shared tasks. in future work, we aim to study the applicability of our ensemble on other geolocation tasks, perhaps taking into consideration future vardial challenges.