Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining (2023)

Abstract: Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance. However, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded performance. Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language. Upon analyzing noise in different languages, we observe that noise types vary greatly across languages. Thus, existing investigations do not generalize trivially to multilingual settings. To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP). RCP combines data augmentation with a contrastive loss term at the pretraining stage and achieves large improvements on noisy (and original test data) across two sentence-level (+3.2%) and two sequence-labeling (+10 F1-score) multilingual classification tasks.

## introduction recently, multilingual pre-trained language models like mbert @xcite , xlm-r @xcite and various others @xcite @xcite have improved multilingual language understanding by pretraining large transformer models on web-scale corpora (such as wikipedia, common-crawl). these models achieve state-of-the-art performance on cross-lingual transfer and many multilingual nlp tasks (wu and dredze, 2019; pires ⋆ equal contribution, ¶ work done while at amazon. @xcite . however, a real-world system will encounter real-world noise, such as linguistic variations and common errors observed in textual data, that are often absent from benchmark datasets. while prior works focused on this issue of robustness in monolingual settings @xcite @xcite , investigation has been scarce for multilingual settings. in this paper, we study the effect of realistic noise in multilingual settings and propose methods to boost the robustness of multilingual language models across four nlp tasks: intent classification (ic), slot labeling (sl), named entity recognition (ner) and natural language inference (nli). due to the lack of multilingual noisy evaluation data, we synthesize benchmarks by mining noise from publicly available corpora and injecting them into the test sets associated with each of the four tasks. we conduct human validation to ensure that this noised data is indeed realistic (see examples from multiatis++ in figure 1 ) and identify the variety of noise-types seen across languages (in §3). these analyses highlight the potential of our testset in evaluating (and motivating future research on) multilingual robustness. to benchmark the performance of multilingual systems, we consider accuracy metrics on two utterance-level tasks (ic% and nli%) and f1scores on two token-level classification tasks (sl-f1 and ner-f1). specifically, we seek to evaluate the model's performance on the noised version of the test datasets in a zero-shot cross-lingual setting. in this scenario, we have training data for a task available only in one language (in our case, english) and test-data in various languages @xcite . while training data augmentation increases model robustness for monolingual (i.e. english) settings, it is not immediately obvious if these robustness gains can transfer across languages, as error types can often be language-specific. for ex- spanish (es) 0.1 96.9% qué aerolíneas vuelan de baltimore a san francesc muéstrame vuelos entr toronto y san diego necesito información de un vuelo y la tarifa de oakland a salt lake city para el jueves antes e sus 8 am de nuevo york a las vegas el domingo con la tarde hindi (hi) 0.05 95.4% मु झे डे ल्टा उड़ानों के बारे में बताइए जो कोच के याित् रयों को नाश्ता दे ता हों मु झे मे िम्फस से लास वे गास तक उड़ान की जरूरत है सोमवार दोपहर ने लॉस एं िजल्स से िपट् सबगर् रिववार दोपहर को िमयामी में क्लीवलैं ड japanese (jp) 0.1 92.3% 来 水曜日にカンザスシティ初シカゴ行き でシカゴ の 午後 7 時ごろ到着して、 り のフライトが木曜日のフライト ワシントン を コロンバス間のすべてのフ ライトの運賃はいくら シャ ロット空港 の 土曜日 err 午后 1 時に 出 する us エア のフライトをリストア ップして 水曜日のフェニックス ミルウォ キ 逝き chinese (zh) 0.1 86.2% 我需要 4 点 后 在 达拉斯起飞飞往旧金山的 联程航班 请列出从纽瓦克飞往 洛杉机 的航班 然而 每天上午 10 点之前从密尔沃基飞往亚 特兰大 拉瓜迪亚 了 豪华轿车服务要多少钱 figure 1: multiatis++ test set injected with real-world noise mined from wikipedia edits. the highest error injection ratio found to be realistic by human experts is shown alongside the realistic utterance percentage. we do not include the noisy test sets for chinese and japanese in our analysis owing to low (< 95%) realism. thus, to improve the robustness of pretrained multilingual models across noise in all languages, we propose robust constrastive pretraining (rcp) that couples multilingual noisy data-augmentation with a contrastive learning loss term during pretraining; this encourages the model to develop similar representations for the original and the noised version of a sentence. on the noisy test sets, our method improves the multilingual model performance across all metrics and multilingual tasks-ic% by 4.9% on multi-atis++, 4.1% on multisnips; sl-f1 by 18.4 on multiatis++, 8.6 on multisnips; ner-f1 by 2.9 on wikiann; nli% by 0.7% on xnli. in summary, our primary contributions are: our code and data is available on github (repo: amazon-science/multilingual-robust-contrastivepretraining) . ## related work many prior works demonstrate the brittleness of neural models on different noise phenomena such as misspellings @xcite @xcite , casing variation @xcite , paraphrases @xcite , morphological variance @xcite , synonyms @xcite , and dialectical variance @xcite . a popular approach to improve the robustness to noise is fine-tuning models with data augmentation @xcite at either the pretraining @xcite or the task-training stage @xcite . these works consider monolingual pre-trained models and primarily focus on english. while recent works on token-free models motivate robustness in multilingual settings @xcite @xcite , examining the robustness of sota multilingual pre-trained models (and improving them) remains unexplored. hence, we investigate-(1) are multilingual models robust to noise seen in different languages (that may be dissimilar to noise types seen in english)? (2) can we get and leverage multi-lingual noise data to improve multilingual models? and (3) do automatic data-augmentation methods designed for english improve robustness to multilingual noise? to boost the robustness of multilingual models to diverse multilingual noise, we leverage multilingual data augmentation at the pretraining stage and use contrastive learning. our effort complements work in computer vision that showcases contrastive learning with adversarial learning at task-training @xcite and pretraining time @xcite can improve model robustness. nlp has also seen a plethora of work that leverages contrastive learning, but seldom to alleviate robustness concerns @xcite . similar concepts, such as adversarial logit pairing @xcite , used at task-training time have proven to be less effective than data augmentation approaches @xcite in boosting robustness. all the aforementioned works lack in at least one of the two novel aspects of this paper-robustness to real-world (as opposed to adversarial) noise, and/or multilinguality. lastly, the aspect of cross-lingual knowledge transfer has been studied in the context of different nlp tasks; typically, from a high-resource language to a low-resource one, as exemplified by the xtreme benchmark @xcite . in this paper, we investigate the cross-lingual transferability of robustness to realworld noise. ## constructing noisy test data as no existing benchmarks exist to evaluate the robustness of multilingual models, we construct noisy test sets in multiple languages for four tasks. first, we construct a word-level error-and-correction dictionary by leveraging the wikipedia edit corpora. then, we sample replacements from this dictionary and inject them into the test data for the various multilingual tasks, focusing on replacements that only affect individual words but do not change word order. finally, we conduct human evaluation to filter out test sets that are not deemed to be realistic by language experts. ## robust contrastive pre-training (rcp) motivation and approach while task-time data augmentation (aka adversarial training) has been effective to boost the robustness of pre-trained models for english, we face two major challenges-(1) lack of supervised multilingual training data in our zero-shot setting, and (2) lack of approaches to synthetically generate noise data for non-english languages. we overcome these with a multilingual data-augmentation approach at pre-training time that uses the multilingual wikipedia edit corpus to expose our models to human errors during pretraining. here, the need of ex-situ injection of noise (for test-data creation §3) is unnecessary as our edit corpus contains pairs of similar sentences, i.e. a version of the sentence before and after revision by a wikipedia contributor ( §3.1). to encourage the model to align the representations of these two sentences in the encoder's output space, we use a contrastive loss term (see figure 3 ). building on previous work on contrastive learning @xcite , robust contrastive pre-training (rcp) considers the original and edited version of a sentence as positive examples and other unrelated sentences as the negative examples. similar to @xcite and reimers and gurevych ( 2019 )), we map variable length sen-tences to fixed-length embeddings with a pooler e @xmath0, where f (•) is a transformer encoder, and g(•) is the mean of the token-level embeddings. given a batch of n (noisy, clean) sentence tuples, we set our original sentence s c as the anchor and the noisy version s n as the corresponding positive pair. 5 other sentences in the batch (i.e. ̸ = s n ) are deemed to be negative examples. we consider the infonce/nt-xent loss @xcite ) for our per-example contrastive loss: we additionally use the standard mlm loss at pretraining time, masking 15% of the input tokens of every sentence (i.e. both noisy and clean) independently. therefore, our final loss function is l mlm-original is the mlm loss on original sentences, and ensures the model does not 'forget' its original pre-training task. l mlm-noisy is the mlm loss on noisy sentences, and can be thought of as dataaugmentation at pre-training time. pre-training details following the domain adaptive pre-training (dapt) approach (gururangan et al., 2020), we start with an existing multilingual pre-trained model and fine tune it with our rcp objective. unlike dapt, we are not interested in specializing in a particular domain, but in increasing robustness to errors. as mentioned before, we use (unfiltered) pairs of correct/incorrect sentences from the multilingual wikipedia archive and include sentences from the lang8 corpus. 6 the lang8 corpora consists of a smaller number of sentences compared to the wikipedia corpus, but proves to be apt for our purpose; it consists of pairs of sentences-one written by a non-native speaker who is learning the language (eg. "as the winter more details about the individual corpora can be found in appendix c. we note that the pre-training corpus is not exactly the same set of sentences used to construct our noise dictionaries in §3.1. in this case, the only criteria for inclusion is a length difference @xmath1, and a relative edit-distance of 30% of the shorter sentence (see appendix c for more details). hence, we incorporate training data from the corpora that exhibit changes beyond simple typos (such as paraphrasing, sentence-level morphological variance) in the pre-training stage. 7 similar to @xcite , we fine tune for 25k steps with a batch size of 2048 sentences to create two pretrained models-one with l contrastive +l mlm-noisy +l mlm-clean (referred to as robust contrastive pre-training or rcp) and an ablation without the contrastive term, i.e. l mlm-noisy + l mlm-clean . the latter setting represents a pure (pre-training time) data augmentation approach such as @xcite (termed p(aug) in ## experiments and results we divide this section into three parts. in §5.1, we analyze the robustness of popular multilingual language models in the zero-shot cross-lingual set7 unfortunately, the benefit of including sentence-level noise in the pre-training phase is not directly examined by our benchmarks, which focus more on word-level noise. ## conclusion in this paper, we investigate the robustness of pretrained multilingual models in the zero-shot crosslingual setting on four tasks-intent classification, slot labeling, named entity recognition, and natural language inference. given the dearth of existing datasets to benchmark the robustness of existing multilingual models, we develop noisy test data by injecting errors mined from an edit corpus (and conduct expert evaluation for quality assurance). our identification of noise types across various languages motivates the necessity of language specific investigation in the future. finally, demonstrate existing baselines perform poorly in the presence of noise in the test data and propose robust contrastive pretraining to boost the robustness of these multilingual models.