Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise (2022)

Abstract: Cross-lingual transfer between a high-resource language and its dialects or closely related language varieties should be facilitated by their similarity. However, current approaches that operate in the embedding space do not take surface similarity into account. This work presents a simple yet effective strategy to improve cross-lingual transfer between closely related varieties. We propose to augment the data of the high-resource source language with character-level noise to make the model more robust towards spelling variations. Our strategy shows consistent improvements over several languages and tasks: Zero-shot transfer of POS tagging and topic identification between language varieties from the Finnic, West and North Germanic, and Western Romance language branches. Our work provides evidence for the usefulness of simple surface-level noise in improving transfer between language varieties.

## introduction recent research has achieved impressive results in zero-shot cross-lingual transfer based on multilingual pre-training @xcite or monolingual transfer of embeddings @xcite . however, these methods require large amounts of unlabeled data in the target language @xcite and do not take into account surface similarity between languages except for the sharing of subword units in multilingual models. for the transfer between closely related languages and dialects, we deem it desirable to exploit the similarity of surface representations. specifically, we target orthographic variations that commonly result from pronunciation differences between closely related languages. 1 1 note that there are also differences on different levels as described in @xcite and partly observable in figure 1 which illustrates a german example sentence with a closely related variant. in this paper, we propose to augment the training data of a high-resource language with characterlevel noise to simulate spelling variations and thus facilitate generalization to closely related foot_0 lowresource languages. we test this strategy on two tasks and several language regions. the considered tasks are part-ofspeech (pos) tagging on the word level and topic classification on the sentence level; the languages are from the finnic, west and north germanic, and western romance language branches. we observe that our baseline method for cross-lingual transfer learns undesirable heuristics, e.g., assigning unseen words to open word classes in pos tagging and that injecting noise reduces this bias. our experiments show absolute accuracy improvements between 1.4 and 22 percentage points over the state of the art, providing evidence that a simple dataaugmentation strategy can boost transfer learning for language varieties and dialects with a closely related high-resource language. artetxe et al. (2020) introduce zero-shot crosslingual transfer by mapping monolingual representations between languages. they also propose adding gaussian noise to the embeddings during the fine-tuning step. huang et al. (2021) also operate in the embedding space by constructing robust regions in the embedding space to tolerate noise in the contextual embedding. these are not ideal strategies for closely related languages because words with similar surface forms could still be far from each other in an embedding space. surface-level noise such as character substitutions, insertions, and deletions has been proposed as an effective data augmentation strategy for machine translation @xcite @xcite @xcite . authors report improvements in system accuracy due to more robustness towards speech recognition errors, spelling mistakes, and other naturally occurring noise in text data. even though cross-lingual transfer between closely related languages has received some attention @xcite @xcite , it has not been investigated whether this transfer can be improved with character-level noise inserted at training time. we tackle this in our work by adding random characterlevel noise to the training data of a standard language and applying the model to closely related languages. exploiting orthographic similarity to improve cross-lingual transfer between closely related languages is currently an understudied area. relevant previous work has been done by @xcite , who used orthographic similarity to refine bilingual dictionary induction. transliteration is another line of related work that focuses on improving the transfer between closely related languages with different alphabets @xcite @xcite . on the other hand, our work focuses on languages using the same script. the recent report by muller et al. figure 2 : methodology for zero-shot cross-lingual transfer: we first continue the pre-training of a language model (lm) on text. then, we fine-tune the adapted lm to task t in high-resource language x. we augment the training data for continued pre-training or fine-tuning with character-level noise and apply the model to task t in a closely related low-resource language y . ## method consider a high-resource language x and a closely related low-resource language y . we perform zeroshot cross-lingual transfer by pre-training a model on unlabeled data from x (and optionally y if available), then fine-tuning on task t in language x. the resulting model is applied to task t in language y . this procedure is illustrated in figure 2 . thus, our question is: how can we best make the model trained on x generalize to y ? ideally, such a model would take surface similarity of words into account for generalization. we hypothesize that in closely related languages, unknown words in the low-resource language are likely to correspond to similar known words in the high-resource language in function and meaning. 3 however, state-of-the-art language models represent words through subwords @xcite . this representation is sensitive to slight surface variations: minor changes to a string will lead to different segmentations and internal representations. to account for this problem, we apply noise to the surface representation of words in language x. we implement this through character-level noise, i.e., we randomly foot_2 select 10% -15% of the tokens of a sentence foot_3 excluding numbers and punctuation. one randomly selected character of the chosen token undergoes one of three possible operations: delete, replace, insert, each with equal probability. the latter two operations work with an additional randomly selected character of the (extended) latin alphabet for the source language. the following sentence this is a short example. will end up with some sort of a "typo", e.g., as this ius a short example. noise can be applied during pre-training on x, and/or during fine-tuning on a task t for language x. another possibility to alleviate the subword representation problem is bpe-dropout @xcite , which applies different segmentations to words in a randomized fashion. bpedropout was originally motivated to increase robustness for morphological variance. we hypothesize that it is similarly effective for orthographic variance; see table a1 for examples. overall, both character-level noise and bpe-dropout encourage the model to learn generalizations across similar surface strings via shared subwords. a second motivation for character-level noise is that we aim to imbue the model with different inductive biases. for example, a model for pos tagging might learn that only a small set of words can map to closed word classes such as articles, whereas unknown words are likely to belong to an open word class such as named entities. training with character-level noise will reduce this bias. ## experiments we design our experimental procedure foot_4 to answer the following question: does character-level noise improve zero-shot transfer to closely related languages? within three controlled experiments, we ablate the importance of the noise-augmentation strategy. we select two cross-lingual tasks: 1) pos tagging (15% noise) and 2) topic classification (10% noise). while the former task illustrates the strategy's potential on word level, the latter provides insight into how much it helps on text level. ## analysis figure 3 illustrates the prediction differences of a model trained with and without noise. we observe that the models trained without noise have learned a tendency towards labeling unknown words as open-class words such as names (ne) or adjectives (adjd), with the label for foreign words (fm) being massively overpredicted, while it tends to undergenerate closed-class tags such as articles (art) or adverbs (adv). in contrast, the model trained with noise comes much closer to the gold standard tag distribution. it has learned to rely more on probable pos tag sequences than on the surface form of a token. consider e.g. the gsw article d (de die; the). in the de training corpus, the token appears only as a foreign word (fm) because it also happens to be a french word, but the model trained with noise is more likely to correctly tag it as an article, relying more on context than just strict mappings. 8 figure 4 depicts the per-type f1 change for the most frequent stts @xcite tags. the past participles of the auxiliary verbs (vapp) form another closed class which profits substantially from a model focusing on tag sequences given the compound structure of the perfect tense. as swiss german does not have a simple past, the perfect tense is much more frequent than it is in german. ## discussion our investigation into cross-lingual zero-shot transfer between closely related languages demonstrates that simple data augmentation with character-level noise can successfully improve transfer, with absolute improvements ranging from 1.4 (@xmath0. the examination of prediction errors shows that a baseline bert model has learned heuristics for unseen words that are undesirable for transfer between closely related languages. in contrast, a model trained with noise can combat this bias without substantial performance losses in the source languages. we expect that the final effectiveness of using character-level noise for zero-shot cross-lingual transfer depends on the task and language characteristics. we plan to evaluate the effect of characterlevel noise in a broader range of settings in future work. more broadly, we encourage further research that exploits surface-level word similarity for crosslingual transfer between related languages and dialects, rather than focusing purely on vector space representations. most frequent error reduction with gsw de en pos without noise correct pos noise (relative/absolute) ond und and fm kon 99.00% (-104) worde geworden become vvpp, adjd vapp 98.73% (-156) dr der the fm, ne, nn art 98.46% (-128) häd hat had vvfin, fm vafin 98.21% (-55) gsi gewesen been vvpp, fm vapp 98.19% (-434) vu von from fm, ne appr 98.18% (-108) eme einem a adja, fm art 97.96% (-48) grad gerade just adjd adv 97.59% (-81) vum vom from the fm, appr apprart 96.49% (-55) de der the fm, ne, adja art 95.76% (-1558) table a1: swiss german (gsw) words (and their corresponding standard german (de) form and english (en) translation) with the highest error reduction using a part-of-speech (pos) tagging model trained with character-level noise compared to the model trained without noise.