Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal A rabic Sequence Labeling (2021)

Abstract: A sufficient amount of annotated data is usually required to fine-tune pre-trained language models for downstream tasks. Unfortunately, attaining labeled data can be costly, especially for multiple language varieties and dialects. We propose to self-train pre-trained language models in zero- and few-shot scenarios to improve performance on data-scarce varieties using only resources from data-rich ones. We demonstrate the utility of our approach in the context of Arabic sequence labeling by using a language model fine-tuned on Modern Standard Arabic (MSA) only to predict named entities (NE) and part-of-speech (POS) tags on several dialectal Arabic (DA) varieties. We show that self-training is indeed powerful, improving zero-shot MSA-to-DA transfer by as large as ˷10% F _1 (NER) and 2% accuracy (POS tagging). We acquire even better performance in few-shot scenarios with limited amounts of labeled data. We conduct an ablation study and show that the performance boost observed directly results from training data augmentation possible with DA examples via self-training. This opens up opportunities for developing DA models exploiting only MSA resources. Our approach can also be extended to other languages and tasks.

## introduction neural language models @xcite with vectorized word representations @xcite are currently core to a very wide variety of nlp tasks. in specific, using representations from transformer-based @xcite language models @xcite , pre-trained on large amounts of unlabeled data and then fine-tuned on labeled taskspecific data, has become a popular approach for improving downstream task performance. this pre-training then fine-tuning scheme has been successfully applied to several tasks, including question answering @xcite , social meaning detection @xcite ), text classification @xcite , named entity recognition (ner), and part-of-speech (pos) tagging @xcite . the same setup also works well for cross-lingual learning @xcite . given that it is very expensive to glean labeled resources for all language varieties and dialects, a question arises: "how can we leverage resourcerich dialects to develop models nuanced to downstream tasks for resource-scarce ones?". in this work, we aim to answer this particular question by applying self-training to unlabeled target dialect data. we empirically show that self-training is indeed an effective strategy in zero-shot (where no gold dialectal data are included in training set, section 4.2) and few-shot (where a given number of gold dialectal data points is included in training split, section 4.4). our few-shot experiments reveal that selftraining is always a useful strategy that consistently improves over mere fine-tuning, even when all dialect-specific gold data are used for fine-tuning. in order to understand why this is the case (i.e., why combining self-training with fine-tuning yields better results than mere fine-tuning), we perform an extensive error analysis based on our ner data. we discover that self-training helps the model most (% = 59.7) with improving false positives. this includes da tokens whose msa orthographic counterparts @xcite are either named entities or trigger words that frequently co-occur with named entities in msa. interestingly, such out-of-msa tokens occur in highly dialectal contexts (e.g., interjections and idiomatic expressions employed in interpersonal social media communication) or ones where the social media context in which the language (da) is employed affords more freedom of speech @xcite and a platform for political satire. we present our error analysis in section 5. context: language use in social media tends to diverge from 'standard', offline norms @xcite . for example, users employ slang, emojis, abbreviations, letter repetitions, and other types of playful practices. this poses a challenge for processing social media data in general. however, there are other challenges specific to arabic that motivate our work. more specifically, we choose arabic to apply our approach since it affords a rich context of linguistic variation: in addition to the standard variety, msa, arabic also has several spoken dialects @xcite @xcite , which differ significantly from the written msa @xcite thus offering an excellent context for studying our problem. arabic dialects differ among themselves and from msa at various linguistic levels: lexical, phonological, morphological, and syntactic. this makes our case much more challenging than that of standard vs. social media english, for example. for a good zero-shot performance in our case, a model is required to accommodate not only lexical distance between msa and da, but also differences in word formation and syntax (related to pos tags, for example) and lexical ambiguity (as the meaning of the same token can vary cross-dialectically). this makes the zeroshot setting even harder, where the performance drops 20% f1 points (see section 4.2). from a geopolitical perspective, arabic also has a strategic significance. this is a function of arabic being the native tongue of 400 million speakers in 22 countries, spanning across two continents (africa and asia) 2 . in addition, the three dialects of our choice, namely egyptian (egy), gulf (glf), and levantine (lev), are popular dialects that are widely used online. this makes our resulting models highly useful in practical situations at scale. pragmatically, ability to develop nlp systems on dialectal tasks with no-to-small labeled dialect data immediately eases a serious bottleneck. arabic dialects differ among themselves and from msa at all linguistic levels, posing challenges to traditional nlp approaches. we also note that our method is 2 https://www.internetworldstats.com/stats19.htm language-independent, and we hypothesize it can be directly applied to other varieties of arabic or in other linguistic contexts for other languages and varieties. tasks: we apply our methods on two sequence labeling tasks, where we have access to both msa and da gold data. in particular, as mentioned above, we perform experiments on pos tagging and ner. each of these tasks has become an integral part of various other nlp applications, including question answering, aspect-based sentiment analysis, machine translation, and summarization, and hence our developed models should have wide practical use. again, we note that our approach itself is task-independent. the same approach can thus be applied to other tasks involving da. we leave testing our approach on other languages, varieties, and tasks for future research. contributions: our work offers the following contributions: we now introduce our method. while the majority of labeled arabic datasets are in msa, most daily communication in the arab world is carried out in da. in this work, we show that models trained on msa for ner and pos tagging generalize poorly to dialect inputs when used in zero-shot-settings (i.e., no dialect data used during training). across the two tasks, we test how self-training would fare as an approach to leverage unlabeled da data to improve performance model labeled examples (msa only at first) confidence scores unlabeled examples (da only) non-selected examples selected examples selection mechanism finetune predict 1 2 select samples 3 on da. as for self-training, it involves training a model using its own predictions on a set of unlabeled data identical from its original training split. our proposed self-training procedure is given two sets of examples: a labeled set l and an unlabeled set u . to perform zero-shot msa-to-da transfer, msa examples are used as the labeled set, while unlabeled da examples are the unlabeled set. as shown in figure 1 , each iteration of the self-training algorithm consists mainly of three steps. first, a pre-trained language model is fine-tuned on the labeled msa examples l. second, for every unlabeled da example u i , we use the model to tag each of its tokens to obtain a set of predictions and confidence scores for each token for a selection mechanism, we experiment with both a thresholding approach and a fixedsize (dong and de melo, 2019) approach. in the thresholding method, a threshold τ is applied on the minimum confidence per example. that is, we only add an example u i to l if min (l see algorithm 1. the fixed-size approach involves, at each iteration, the selection of the top s examples with respect to the minimum confidence score min (l ## experiments we begin our experiments with evaluating the standard fine-tuning performance of xlm-r models on both ner and pos tagging against strong baselines. we then use our best models from this first round to investigate the msa-to-da zeroshot transfer, showing a significant performance drop even when using pre-trained xlm-r. consequently, we employ self-training for both ner and pos tagging in zero-and few-shot settings, showing substantial performance improvements in both cases. we now introduce our datasets. ## error analysis to understand why self-training the pre-trained language model, when combined with fine-tuning, improves over mere fine-tuning, we perform an error analysis. for the error analysis, we focus on the ner task where we observe a huge selftraining gain. we use the development set of twitter-da (see section 4.3) for the error analysis. we compare predictions of the standard finetuned xlm-r b model (ft) and the best performing self-training (τ = 0.9) model (st) on the data, and provide the confusion matrices of both models with gold labels in in other words, self-training helps regularize the model predictions such that tokens misclassified by the original ft model as a named entities are now correctly tagged as unnamed entity "o". to understand why the st model improves false positive rate, we manually inspect the cases it cor- in row (3) in we also investigate errors shared by both the ft and st models (errors which the st model also could not fix). some of these errors result from the fact that often times both msa and da use the same word for both person and location names. row (1) in false negatives. the "regularizing" effect caused by self-training we discussed thus far can sometimes produce false negatives as shown in pre-trained language models. language models, based on transformers @xcite , and pre-trained with the masked language modeling (mlm) objective have seen wide use in various nlp tasks. examples include bert @xcite , roberta @xcite , mass @xcite , and electra @xcite . while they have been applied to several tasks, including text classification, question answering, named entity recognition @xcite , and pos tagging @xcite , a sufficiently large amount of labeled data is required for good performance. concurrent with our work, abdul-mageed et al. (2020a) released mar-bert, a language model trained on a large amount of dialectal arabic data. however, the extent to which dialect-specific models such as marbert can alleviate lack of labeled data remains untested. cross-lingual learning. cross-lingual learning is of particular importance due to the scarcity of labeled resources in many of the world's languages. the goal is to leverage existing labeled resources in high-resource languages (such as english) to optimize learning for low-resource ones. in our case, we leverage msa resources for building da models. with proximity to our work, @xcite multilingual extensions of lms have emerged through joint pre-training on multiple languages. examples include mbert @xcite , xlm @xcite and xlm-roberta @xcite . such multilingual models have become useful for few-shot and zero-shot cross-lingual settings, where there is little or no access to labeled data in the target language. for instance @xcite evaluated a cross-lingual version of roberta @xcite , namely xlm-r, on cross-lingual learning across different tasks such as question answering, text classification, and named entity recognition. ## conclusion even though pre-trained language models have improved many nlp tasks, they still need labeled data for fine-tuning. we show how self-training can boost the performance of pre-trained language models in zero-and few-shot settings on various arabic varieties. we apply our approach to two sequence labeling tasks (ner and pos), establishing new state-of-the-art results on both. through in-depth error analysis and an ablation study, we uncover why our models work and where they can fail. our method is languageand task-agnostic, and we believe it can be applied to other tasks and language settings. we intend to test this claim in future research. our research also has bearings to ongoing work on language models and self-training, and interactions between these two areas can be the basis of future work. all our models and code are publicly available.