Combining Deep Learning and String Kernels for the Localization of S wiss G erman Tweets (2020)

Abstract: In this work, we introduce the methods proposed by the UnibucKernel team in solving the Social Media Variety Geolocation task featured in the 2020 VarDial Evaluation Campaign. We address only the second subtask, which targets a data set composed of nearly 30 thousand Swiss German Jodels. The dialect identification task is about accurately predicting the latitude and longitude of test samples. We frame the task as a double regression problem, employing a variety of machine learning approaches to predict both latitude and longitude. From simple models for regression, such as Support Vector Regression, to deep neural networks, such as Long Short-Term Memory networks and character-level convolutional neural networks, and, finally, to ensemble models based on meta-learners, such as XGBoost, our interest is focused on approaching the problem from a few different perspectives, in an attempt to minimize the prediction error. With the same goal in mind, we also considered many types of features, from high-level features, such as BERT embeddings, to low-level features, such as characters n-grams, which are known to provide good results in dialect identification. Our empirical results indicate that the handcrafted model based on string kernels outperforms the deep learning approaches. Nevertheless, our best performance is given by the ensemble model that combines both handcrafted and deep learning models.

## introduction the organizers of the 2020 vardial evaluation campaign @xcite proposed a shared task targeted towards the geolocation of short texts, e.g. tweets, namely the social media variety geolocation (smg) task. typically formulated as a double regression problem, the task is about predicting the location, expressed in latitude and longitude, from where the text received as input was posted on a certain social media platform. twitter and jodel are the platforms used for data collection, divided by the language area in three subtasks, namely: in this paper, we focus only on the second subtask, smg-ch, proposing a variety of handcrafted and deep learning models, as well as an ensemble model that combines all our previous models through meta-learning. our first model is a support vector regression (svr) classifier @xcite based on string kernels, which are known to perform well in other dialect identification tasks @xcite @xcite . our second model is a character-level convolutional neural network (cnn) @xcite , which is also known to provide good results in dialect identification @xcite . due to the high popularity and the outstanding results of bidirectional encoder representations from transformers (bert) @xcite in solving mainstream nlp tasks, we decided to try out a long short-term memory (lstm) network @xcite based on german bert embeddings as our third model. lastly, we combine our three models into an ensemble that employs extreme gradient boosting (xgboost) @xcite as meta-learner. we conducted experiments on the development set provided by the organizers, in order to decide which models to choose for our three submissions for the smg-ch subtask. our results indicate that the ensemble model attains the best results. perhaps surprisingly, our shallow approach based on string kernels outperforms both deep learning models. our observations are consistent across the development and the test sets provided by the organizers. the rest of this paper is organized as follows. we present related work on dialect identification and geolocation of short texts in section 2. our approaches are described in more detail in section 3. we present the experiments and empirical results in section 4. finally, our conclusions are drawn in section 5. ## related work one of the initial works on text-based geotagging @xcite aims at automatically finding the geographic scope of web pages, in a classification setup relying on named location entities such as cities and states. the authors used gazetteers as the source of the location mappings, proposing a rather heuristic approach. gazetteers, constitute a tool used in one of the three general approaches taken so far in text-based geolocation, this tool being adopted in a number of works @xcite @xcite . in this line of research, some researchers employed rule-based methods @xcite , while others plugged named entity recognition into various machine learning techniques @xcite . the main disadvantage of these methods is that they rely on the existence of specific mentions of locations in text, rather than inferring them in a not so straightforward manner. these direct mentions of places do not represent a safe assumption, especially when it comes to social media platforms such as twitter, which is used as the data source in some of these studies @xcite . the other two main categories of approaches for text-based geolocation rely on either unsupervised learning @xcite @xcite or supervised classification @xcite . the unsupervised methods can be described in large part as clustering techniques based on topic models. there are some studies on user geolocation in social media, that look at this task from a supervised learning perspective @xcite and can be included in the second set of approaches for geotagging. however, in such works, other details (e.g. social ties) in the users profile have been considered rather than their written content. although these works cover geolocation prediction in social media, they do not use text as input. our current interest in studying language variation for the geolocation of users in social media has been covered in the literature in a series of works @xcite @xcite @xcite , employing various machine learning techniques, that range from probabilistic graphical models @xcite and adaptive grid search @xcite to bayesian methods @xcite and neural networks @xcite . the related work to date covers a wide range of languages and dialects, including dutch @xcite , british @xcite , american @xcite and even african american vernacular english @xcite . most related to our work is the study of @xcite , which targets the german language and its variations and, in addition to the previously mentioned endeavours, performs a quantitative analysis against a dialect map. moreover, hovy and purschke (2018) collected 16.8 million online posts from the german-speaking area with the aim of learning document representations of cities. among these posts, some were from the german speaking side of switzerland, being part of the smg shared task, more specifically the smg-ch subtask that we are addressing. the authors aimed at capturing enough regional variations in the written language, serving as input in automatically distinguishing the geographical region of speakers. the focus was on larger regions covering a given dialect, the proposed approach being based on clustering. given the shared task formulation, we take a different approach and use the provided data in a double regression setup, addressing the problem both from a shallow perspective and a deep learning perspective, respectively. ## conclusion in the current work, we tackled the smg-ch shared subtask of the 2020 vardial evaluation campaign. we addressed this challenge from a shallow perspective, with handcrafted models such as a ν-svr based on string kernels, as well as from a deep learning perspective, with neural models such as an lstm based on bert embeddings and a character-level cnn, respectively. additionally, we combined the proposed models into an ensemble, employing the xgboost meta-learner. we obtained our best results with the xgboost ensemble, which benefits from complementary information from the handcrafted and deep models. we therefore brought one more proof regarding the effectiveness of ensemble learning in general, and of xgboost, in particular. another important conclusion is that our shallow model based on string kernels outperforms the two deep neural networks. we consider this as yet another indicator of the high discriminative power that string kernels can bring to a fairly standard learning model, i.e. the ν-svr. in future work, we aim to explore ways to improve our performance with respect to the metrics proposed by the shared task organizers. currently, it seems that training the models to simply minimize the mse or the mae values is not effective, as our best model was significantly outperformed by the model proposed by the shared task organizers themselves.