SLICER : Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition (2022)

Abstract: Large multilingual language models generally demonstrate impressive results in zero-shot cross-lingual transfer, yet often fail to successfully transfer to low-resource languages, even for token-level prediction tasks like named entity recognition (NER). In this work, we introduce a simple yet highly effective approach for improving zero-shot transfer for NER to low-resource languages. We observe that NER fine-tuning in the source language decontextualizes token representations, i.e., tokens increasingly attend to themselves. This increased reliance on token information itself, we hypothesize, triggers a type of overfitting to properties that NE tokens within the source languages share, but are generally not present in NE mentions of target languages. As a remedy, we propose a simple yet very effective sliced fine-tuning for NER (SLICER) that forces stronger token contextualization in the Transformer: we divide the transformed token representations and classifier into disjoint slices that are then independently classified during training. We evaluate SLICER on two standard benchmarks for NER that involve low-resource languages, WikiANN and MasakhaNER, and show that it (i) indeed reduces decontextualization (i.e., extent to which NE tokens attend to themselves), consequently (ii) yielding consistent transfer gains, especially prominent for low-resource target languages distant from the source language.

## introduction in recent years, massively multilingual transformers (mmts) have become the backbone of multilingual nlp. mmts like mbert @xcite , xlm-r @xcite , and mt5 @xcite , pretrained on corpora spanning 100+ languages, have become the main vehicle of cross-lingual transfer in nlp: fine-tuned using task annotations in the source language, an mmt can, conceptually, directly make predictions for all target languages seen in pretraining, for which little or no annotated task data exists @xcite @xcite . successful zero-shot transfer, however, has been shown to critically hinge on linguistic proximity between source and target languages as well the quality of target language representations, determined by the size of target language corpora used in the mmt pretraining @xcite . unfortunately, the transfer fails where it is needed the most -for low-resource languages linguistically distant from high-resource languages with annotated task data @xcite @xcite . zeroshot transfer of named entity recognition (ner) models to low-resource languages suffers from a particularly profound performance drop @xcite . in this work, we identify the cause and propose an effective remedy. ## evaluation experimental setup. unless stated differently, we train on the english training portion of wikiann @xcite . we then evaluate slicer against standard fine-tuning (standard ft) in zero-shot transfer to (i) 23 target languages from wikiann and (ii) 10 african languages from masakhaner @xcite . 5 we finetune xlm-r (base) with mixed precision, using adamw with 0.05 weight decay (loshchilov and hutter, 2019) for optimization. we train with three different learning rates {5e -6 , 1e -5 , 2e -5 } with 10% linear warmup and subsequent decay, for each setup of which we execute 10 fine-tuning runs with different random seeds. we apply 10% dropout and train in batches of size 32 for 10 epochs. we compare slicer against standard ft under two common evaluation procedures for crosslingual transfer: (1) true zero-shot transfer strictly assumes that there are no labeled instances in the target language; (2) oracle transfer assumes that a small validation set in the target language is available for model selection: in this setting, we select the model checkpoint that yields the best target language validation performance. for slicer, we report the results for three different values of slice size, @xmath0, 2, 8}. results. ## conclusion in this focused research effort, we show that (monolingual) fine-tuning for ner introduces token decontextualization in higher transformer layers which, we hypothesize, has a negative effect on (zero-shot) cross-lingual ner transfer with mmts. we devise a novel sliced fine-tuning approach, dubbed slicer, that reduces this decontextualization effect by splitting transformed token vectors into disjoint slices which are then independently classified. we demonstrate on wikiann and masakhaner that this yields substantial transfer gains, especially when transferring to low-resource languages. we additionally show that gains do not stem from a particular choice of source language. our work shows that, despite the task-agnostic nature of the predominant mmt-based cross-lingual transfer paradigm, task specificities can still be leveraged to improve the cross-lingual transfer.