Geographic Adaptation of Pretrained Language Models (2024)

Abstract: While pretrained language models (PLMs) have been shown to possess a plethora of linguistic knowledge, the existing body of research has largely neglected extralinguistic knowledge, which is generally difficult to obtain by pretraining on text alone. Here, we contribute to closing this gap by examining geolinguistic knowledge, i.e., knowledge about geographic variation in language. We introduce geoadaptation, an intermediate training step that couples language modeling with geolocation prediction in a multi-task learning setup. We geoadapt four PLMs, covering language groups from three geographic areas, and evaluate them on five different tasks: fine-tuned (i.e., supervised) geolocation prediction, zero-shot (i.e., unsupervised) geolocation prediction, fine-tuned language identification, zero-shot language identification, and zero-shot prediction of dialect features. Geoadaptation is very successful at injecting geolinguistic knowledge into the PLMs: The geoadapted PLMs consistently outperform PLMs adapted using only language modeling (by especially wide margins on zero-shot prediction tasks), and we obtain new state-of-the-art results on two benchmarks for geolocation prediction and language identification. Furthermore, we show that the effectiveness of geoadaptation stems from its ability to geographically retrofit the representation space of the PLMs.

## introduction the default tool for the majority of nlp tasks is now de facto pretrained language models (plms; @xcite @xcite @xcite @xcite @xcite , which are trained using language modeling objectives on large text corpora. despite the conceptual simplicity of language modeling, pretraining induces complex forms of linguistic knowledge in plms, at various levels @xcite : morphological @xcite @xcite , lexical @xcite , syntactic @xcite @xcite , and semantic @xcite . this general linguistic knowledge is then (re-)shaped for concrete tasks via fine-tuning, i.e., supervised training on task-specific labeled data. humans, however, additionally make use of a rich spectrum of extralinguistic features when they learn and process language, including gender @xcite , ethnicity @xcite , and geography @xcite . despite the growing awareness for the importance of such factors in nlp @xcite , extralinguistic features have been typically introduced in the fine-tuning phase so far, i.e., when specializing plms for a concrete task (e.g., @xcite . this prevents plms from forming generalizable representations the way humans do, impeding the exploitation of extralinguistic knowledge for tasks other than the fine-tuning task itself. in this work, we focus on geographic knowledge, and more specifically geolinguistic knowledge, i.e., knowledge about geographic variation in language-the most salient type of extralinguistic variation in language @xcite . we present what we believe to be the first attempt to incorporate geolinguistic knowledge into plms in a pretraining step, i.e., before task-specific fine-tuning, making it possible to exploit it in any task for which it is expected to be useful. specifically, we conduct an intermediate training step @xcite in the form of task-agnostic adaptation-dubbed geoadaptation-that couples language modeling with predicting the geographic location (i.e., longitude and latitude) on geolocated texts. we choose adaptation as opposed to pretraining from scratch for three reasons: (i) intermediate training on language modeling (i.e., adaptation) before task-specific fine-tuning has proved beneficial for many nlp tasks @xcite , (ii) adaptation has a lower computational cost than pretraining @xcite , and (iii) plms encoding general-purpose linguistic knowledge are readily available @xcite . 1 the specific method we introduce for geoadaptation combines language modeling with token-level geolocation prediction via multi-task learning, with task weights based on the homoscedastic uncertainties of the task losses @xcite . we evaluate our geoadaptation framework on three groups of closely related languages, each with a corresponding plm: (i) the german dialects spoken in austria, germany, and switzerland (ags) and germanbert; (ii) bosnian-croatian-montenegrin-serbian (bcms) and bertić; and (iii) danish, norwegian, and swedish (dns) and scandibert. these groups exhibit strong geographic differences, providing an ideal testbed for geoadaptation. 2 we further test geoadaptation at scale by adapting mbert, a multilingual plm, on the union of ags, bcms, and dns. we evaluate the effectiveness of geoadaptation on five downstream tasks expected to benefit from geolinguistic knowledge: (i) fine-tuned (i.e., supervised) geolocation prediction, (ii) zero-shot (i.e., unsupervised) geolocation prediction, (iii) fine-tuned language identification, (iv) zero-shot language identification, and (v) zero-shot prediction of dialect features. geoadaptation leads to consistent performance gains compared to baseline models adapted on the same data using only language modeling, with particularly striking improvements on all zero-shot tasks. on two popular benchmarks for geolocation prediction and language identification, geoadaptation establishes a new state of the art. furthermore, we show that geoadaptation geographically retrofits the representation space of the plms. overall, we see our study as an exciting step towards grounding plms in geography. 3 2 related work adaptation of plms. continued language modeling training (i.e., adaptation) on data that comes from a similar distribution as the task-specific target data has been shown to improve the performance of plms for many nlp tasks @xcite as well as in various language @xcite and domain adaptation scenarios @xcite . adaptation can be seen as a special case of intermediate training, which aims at improving the target-task performance of plms by carrying out additional training between pretraining and fine-tuning @xcite @xcite . intermediate training has also been conducted in a multi-task fashion, encompassing two or more training objectives @xcite . our work differs from these efforts in that it injects geolinguistic knowledge-a type of extralinguistic knowledge-into plms. ## geoadaptation let d be a geotagged dataset consisting of sequences of tokens @xmath0, . . . , x n ) and corresponding geotags @xmath1, t lat ), where t lon and t lat denote the geographic longitude and latitude. we want to adapt a plm in such a way that it encodes the geographically conditioned linguistic variability in d. acknowledging the prominence of lexical variation among geographic differences in language (see §2), we accomplish this by combining masked language modeling (i.e., the pretraining objective) with token-level geolocation prediction in a multi-task setup that pushes the plm to learn associations between linguistic phenomena and geolocations on the lexical level. 4 masked language modeling. we replace some tokens x i in x with masked tokens xi . following @xcite , xi can be a special mask token ([mask]), a random vocabulary token, or the original token itself. x is fed into the plm, which outputs a sequence of representations @xmath2, . . . , e(x n )). the representations of the masked tokens e(x i ) are then fed into a classification head. we compute the masked language modeling loss l mlm as the negative log-likelihood of the probability assigned to the true token. ## results and analysis tables 3, 5, 6, and 7 compare the performance of the geoadapted plms against the baselines. to test for statistical significance of the performance differences, we use paired, two-sided student's t-tests in the case of ft-geoloc and mcnemar's tests for binary data @xcite in the case of zs-geoloc, ft-lang, zs-lang, and zs-dialect, as recommended by @xcite . we correct the resulting p-values for each evaluation using the holm-bonferroni method @xcite . overall, the geoadapted models consistently and substantially outperform the baselines-out of the 30 main evaluations, it is always one of the two geoadapted models that achieves the best score, a result that is highly unlikely to occur by chance if there is no underlying performance difference between the geoadapted and non-geoadapted models. 16 furthermore, in the two cases where we can directly compare to a prior state of the art, one or both geoadapted models outperform it. these findings strongly suggest that geoadaptation successfully induces associations between language variation and geographic location. 16 assuming equal underlying performance for mlmada, geoada-s, and geoada-w (and ignoring other baselines), the probability for this result is @xmath3. ## conclusion we introduce geoadaptation, an approach for task-agnostic continued pretraining of plms that forces them to learn associations between linguistic phenomena and geographic locations. the method we propose for geoadaptation couples language modeling and token-level geolocation prediction via multi-task learning. while we focus on plms pretrained via masked language modeling, geoadaptation can in principle be applied to autoregressive plms as well. we geoadapt four plms and obtain consistent gains on five tasks, establishing a new state of the art on established benchmarks. we further show that geoadaptation acts as a form of geographic retrofitting. overall, we see our study as an exciting step towards nlp technology that takes into account extralinguistic aspects in general and geographic aspects in particular.