Fine-Tuning BERT with Character-Level Noise for Zero-Shot Transfer to Dialects and Closely-Related Languages (2023)

Abstract: In this work, we induce character-level noise in various forms when fine-tuning BERT to enable zero-shot cross-lingual transfer to unseen dialects and languages. We fine-tune BERT on three sentence-level classification tasks and evaluate our approach on an assortment of unseen dialects and languages. We find that character-level noise can be an extremely effective agent of cross-lingual transfer under certain conditions, while it is not as helpful in others. Specifically, we explore these differences in terms of the nature of the task and the relationships between source and target languages, finding that introduction of character-level noise during fine-tuning is particularly helpful when a task draws on surface level cues and the source-target cross-lingual pair has a relatively high lexical overlap with shorter (i.e., less meaningful) unseen tokens on average.

## introduction contemporary nlp methods such as bert @xcite , with the large amount of knowledge contained within their parameters, paired with the relatively low computational power required to finetune them for a downstream task, have taken over many nlp applications. indeed, several monolingual and multilingual bert models are available that encompass a number of languages @xcite . however, the strength of these models is tied to the availability of data, and the large amounts of data required to pre-train such models exclude some languages for which it is difficult to collect large amounts of written text. the scarcity of data becomes more severe with dialects and language varieties. in fact, the very nature of dialects as an evolving form of the language, often spoken rather than written, with various social and cultural nuances, can make it difficult to develop systems tailored to specific dialects. in many applications, users may span a continuum of idiolects, some falling into established dialects and others not. it may therefore be impossible to train a system on even a small amount of data in every idiolect. in this paper, we consider zero-shot cross-lingual transfer, which we define strictly as any scenario in which the test data is of a language variety not included in any stage of training. for instance, we may fine-tune a standard italian bert model on standard italian sentiment analysis data, and then perform inference on neapolitan (a variety closely related to standard italian). we call standard italian the source language, and neapolitan the target language. the mismatch in bert's performance when evaluating on the source versus target language can arise for a variety of reasons depending on the properties of the target language. for example, some language varieties may have similar morphology but different vocabulary, so that bert may encounter completely new words when tested on the dialect. an example of this is the use of "soda" in some regions of the united states and "pop" in others to refer to a carbonated beverage; a model trained only on the "soda" varieties may have difficulty identifying the meaning of "pop" if it appears in test data. in other cases, language varieties may have similar vocabulary, but phonological, morphological, or orthographic differences may throw off the subword tokenization method used by the model. a simple example is the distinction in spelling between "color" (american english) and "colour" (british english); if a model were trained exclusively on american english and "colour" was not part of its vocabulary, at test time, "colour" would be tokenized differently than "color," possibly resulting in a different interpretation by the model. in this work, we focus on the second type of dialectal variation. following @xcite , we study how introducing character-level noise in training can improve performance for zeroshot cross-lingual transfer between closely-related languages. @xcite method is a two-step process. the first step is continued pre-training of bert on three types of data: target language data, un-noised source language data, and noised source language data. the second step is fine-tuning on noised task data in the source language. here, we only use fine-tuning, and we only use source-language data, making our method strictly zero-shot. we explore the next questions: which techniques of character-level noising help cross-lingual transfer the most, and in which situations should one expect character-level noising to work best? to explore these questions, we fine-tune monolingual bert models on three sentence-level classification tasks: intent classification, topic identification, and sentiment analysis. we introduce multiple variations on the method of noising in order to optimize cross-lingual transfer. we test our methods on an assortment of unseen languages, some closely-related and some more distant relatives. for the intent classification task, our systems work almost perfectly, that is, they perform nearly as well on the target language as on the source language. we also boost task performance in less closely-related languages (in the same and different families). furthermore, we find that we can obtain even bigger improvements by using more noise, and we find that exposing the model to more variations of the data during fine-tuning also helps. finally, we explore the conditions for cross-lingual transfer needed for our method to be successful. ## methods our experiments focus on fine-tuning monolingual language models on same-language data, and testing for zero-shot cross-lingual transfer to other languages, inducing noise in the fine-tuning data to facilitate this transfer. building on @xcite promising finding that characterlevel noise can be used as a conduit for crosslingual transfer between closely-related languages, we introduce a range of options for applying character noise in order to explore how we can better leverage the benefits of character-level noise for zero-shot cross-lingual transfer. ## experiments in order to evaluate the effectiveness of inducing character-level noise for zero-shot cross-lingual transfer under the various settings described in section 3.3, we test on three tasks: intent classification, topic identification, and sentiment analysis. all three tasks are sentence-level classification tasks; however, each task has unique challenges that can bolster or break compatibility with our approach. we are interested in seeing how noise can help in each of these scenarios. ## conclusion in this work, we explore two questions: first, when is it a good idea to use character-level noise in finetuning as an agent for zero-shot cross-lingual transfer, and second, in cases where inducing characterlevel noise is helpful, which noising techniques work the best? we fine-tune monolingual bert models on three sentence-level classification tasks, each with a different source language, introducing several variations in the method of noising for the fine-tuning data. we test on a medley of unseen dialects, closely-related languages, and distant relatives. we find that one of our test settings lends itself particularly well to our method, while the other two do not. this distinction comes down to the nature of the task and the relationship (in terms of lexical overlap) between the cross-lingual source-target pair tested. our extensions in the space of noising variations allow us to optimize zero-shot cross-lingual transfer to the unseen target languages for the the success case, yielding a boost in performance not only for closely-related pairs, but also for more distant pairs.