Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages (2023)

Abstract: One of the challenges with finetuning pretrained language models (PLMs) is that their tokenizer is optimized for the language(s) it was pretrained on, but brittle when it comes to previously unseen variations in the data. This can for instance be observed when finetuning PLMs on one language and evaluating them on data in a closely related language variety with no standardized orthography. Despite the high linguistic similarity, tokenization no longer corresponds to meaningful representations of the target data, leading to low performance in, e.g., part-of-speech tagging. In this work, we finetune PLMs on seven languages from three different families and analyze their zero-shot performance on closely related, non-standardized varieties. We consider different measures for the divergence in the tokenization of the source and target data, and the way they can be adjusted by manipulating the tokenization during the finetuning step. Overall, we find that the similarity between the percentage of words that get split into subwords in the source and target data (the split word ratio difference) is the strongest predictor for model performance on target data.

## introduction transformer-based pre-trained language models (plms) enable successful cross-lingual transfer for many natural language processing tasks. however, the impact of tokenization and its interplay with transferability across languages, especially under-resourced variants with no orthography, has obtained limited focus so far. tokenization splits words into subwords, but not necessarily in a meaningful way. an example with a current plm is illustrated for alsatian german in figure 1a . this problem is especially pronounced for vernacular languages and dialects, where words tend to be split at a much higher rate than the standard. this has been observed on, e.g., informally written algerian arabic @xcite . as poor a. m'r redd alemànnischi mundàrte . m, ', r red, ##d al, ##em, ##à, mund, . ##nn, ##isch, ##i ##à, ##rte b. wir sprechen alemannische mundarten . wir sprechen al, #emann, mund, ##arten . ##ische c. w(r sprechen alemainische mundarten . w, (, r sprechen al, ##ema, ##in, mund, ##arten . ## method we make our code, including scripts for reproducing the benchmark, available at github.com/ mainlp/noisydialect. ## results and discussion all results we report are averaged over five different random initializations. zero-shot transfer. performance on the unseen test languages/dialects is much lower than on the test partitions of the corpora on which the models were finetuned. this is expected, as there are not only orthographic and stylistic differences between the corpora, but also some grammatical differences between the language varieties. the extent to which performance drops is language-dependent: for instance, the best results for the finnish dialects are 12-17 percentage points below the best results for the finnish standard language (xlm-r), whereas the best results for the norwegian dialects are 26-29 percentage points below the standard language accuracy (nynorsk with norbert). when we have multiple target dialects for one source language, the target scores tend to be similar to one another across noise levels and plm choices. plm choice matters for low-resource languages. while the models are for the most part indistinguishable in their performance on the source languages, the performance on the target languages can vary substantially. for instance, xlm-r outperforms mbert and finbert on the finnish dialect data. similarly, both multilingual models perform much better than the monolingual models on the low saxon, picard and occitan data, and the reverse is true for the arabic dialects. neither the performance on the source languages nor the transfer performance with @xmath0. ## conclusion we have confirmed the usefulness of the noise injection method by @xcite for model transfer between closely related languages. to that end, we have converted additional dialectal datasets to the upos standard and make the conversion code available to other researchers. furthermore, we have shown that the ideal amount of noise that should be injected at finetuning time depends on the languages and plms used. we have also investigated the role that subword tokenization plays in this and found that the split word ratio difference -the (absolute) difference between the proportion of words split into subword tokens in the source and target data -is a reliable, albeit imperfect, predictor of the performance of the transfer model.