S elect N oise: Unsupervised Noise Injection to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages (2023)

Abstract: In this work, we focus on the task of machine translation (MT) from extremely low-resource language (ELRLs) to English. The unavailability of parallel data, lack of representation from large multilingual pre-trained models, and limited monolingual data hinder the development of MT systems for ELRLs. However, many ELRLs often share lexical similarities with high-resource languages (HRLs) due to factors such as dialectical variations, geographical proximity, and language structure. We utilize this property to improve cross-lingual signals from closely related HRL to enable MT for ELRLs. Specifically, we propose a novel unsupervised approach, \textit{SelectNoise} , based on \textit{selective candidate extraction} and \textit{noise injection} to generate noisy HRLs training data. The noise injection acts as a regularizer, and the model trained with noisy data learns to handle lexical variations such as spelling, grammar, and vocabulary changes, leading to improved cross-lingual transfer to ELRLs. The selective candidates are extracted using BPE merge operations and edit operations, and noise injection is performed using greedy, top-p, and top-k sampling strategies. We evaluate the proposed model on 12 ELRLs from the FLORES-200 benchmark in a zero-shot setting across two language families. The proposed model outperformed all the strong baselines, demonstrating its efficacy. It has comparable performance with the supervised noise injection model. Our code and model are publicly available.

## introduction the modern neural machine translation (nmt; @xcite ; @xcite ; @xcite ) has achieved remarkable performance for many languages, but their performance heavily relies on the availability of large parallel or monolingual corpora @xcite . however, the linguistic diversity across the world is vast, with over 7000 languages spoken foot_0 . this linguistic landscape includes a long tail of languages @xcite that face significant challenges due to the lack of available resources for model development and are referred as extremely low-resource languages (elrls). elrls present unique challenges for the development of mt systems as they lack parallel datasets, are excluded from large multilingual pre-trained language models, and possess limited monolingual datasets. consequently, the majority of research efforts in nmt have primarily focused on resource-rich languages @xcite , leaving elrls with limited attention and fewer viable solutions. towards these concerns, this work is positioned as a step towards enabling mt technology for elrls. primarily focused on zero-shot setting for scalability. more recently, there has been active research to develop mt systems for lrls. one direction is multilingual training-based models @xcite @xcite . these models are trained with multiple hrl languages, enabling cross-lingual transfer capabilities to improve translation performance for lrls. another line of work focuses on data augmentation techniques @xcite to generate more training data. however, these methods do not fully exploit the lexical similarity between hrls and elrls. many hrls and elrls exhibit surface-level lexical similarities due to dialect variations, loan words, and geographical proximity @xcite . for example, the word "monday" is somvar in hindi and somar in bhojpuri. they are lexically very similar. to leverage this lexical similarity, recent studies have explored techniques like learning overlapping vocabulary @xcite or injecting random noise @xcite in hrls to resemble lrls. these methods are only evaluated for natural language understanding tasks (nlu) and not for mt, which is a more challenging task. inspired by these advancements, in this paper, we propose a novel unsupervised noise injection approach to develop an mt system for elrls. the proposed model is based on character noise injection and consists of two stages: selective candidate extraction and noise injection. in the selective candidate extraction phase, candidate characters are extracted in an unsupervised manner using small monolingual data from closely related hrl and elrls. it relies on bpe merge operations and edit operations that take into account lexical similarity and linguistic properties. in the noise injection phase, noise is injected into the source side of parallel data of hrl using greedy, top-k, and top-p sampling algorithms. this noise injection serves as a regularizer and a model trained with this noisy hrl data enhances robustness to spelling, grammar, or vocabulary changes and facilitates improved cross-lingual transfer for elrls. the evaluation is done in the zero-shot setting, ensuring scalability. the proposed model is referred as the selectnoise: unsupervised selective noise injection model. fig. 1 illustrates the effect of noise injection with selectnoise model. in this paper, we investigate two hypotheses: (a) the selective noise injection model is expected to outperform random noise injection, and (b) the performance of the selective (unsupervised) noise injection model should be comparable to the supervised noise injection model. our key contributions are: (1) we propose a novel unsupervised selective character noise injection approach, selectnoise, to enable and improve mt for elrls to english. the injection of selective candidate noise facilitates better cross-lingual transfer for elrls in the zero-shot setting. (2) we have developed an unsupervised mechanism to extract candidate characters based on bpe merge operations and edit operations. furthermore, the noise injection employs greedy, top-k, and top-p sampling techniques, ensuring diversity. ## methodology this section presents the details of the proposed selectnoise model. as discussed in section 1, the selectnoise model has two stages: selective candidate extraction and noise injection. in the selective candidate extraction stage, the noise injection candidate characters are extracted through an unsupervised approach. specifically, we consider small monolingual data for hrl (d h ) and for related (lexically similar) lrls (d l ). d l comprises small monolingual datasets from multiple extremely elrls. during the process of building the byte pair encoding (bpe) vocabulary, we extract the bpe operations separately for the hrl (b h ) and elrls (b l ). next, we design an algorithm a to extract selective candidate characters from b h and b l and store them in candidate pool s c . candidates are extracted with an approach inspired by edit-operations. in other words, we obtain s c as a result of a (b h , b l ). in noise injection stage, the candidates are sampled from s c and injected into the source sentences of hrl corpus (h) from the large parallel data en}. the injections are done using a noise function η, resulting in a noise-injected (augmented) parallel data: @xmath0, e)|lang( ĥ) = ĥ, lang(e) = source hrl hrl learn vocabulary transformers encoder transformers decoder source noisy hrl target hrl source elrl zero-shot generation in target (a) model training (b) model inference trained model (m') selective character candidate pool: bpe merge operations bpe merge operations extraction of edit-operations candidate sampling: greedy, top-k, top-p transformers encoder transformers decoder elrl 2 -hne elrl n -awa ... elrl 1 -bho closely related figure 2: overview of the proposed selectnoise model for extremely low-resource mt en}, where ĥ = η(h). the ĥ acts as proxy training data for elrls. a bpe vocabulary v is learned with ĥ. then, we train the standard encoder-decoder transformers model (m; vaswani et al. (2017b) from scratch with ĥ and v to obtain trained model m. finally, zero-shot evaluation is done for elrls with m and v. now, we present details of each model component. the overview of the proposed selectnoise model is depicted in figure 2 . ## experimental setup we designed our experimental setup to address the following set of questions: (1) does noise injection improve performance for nlg tasks, i.e., mt in our case? (2) does selective noise injection with the proposed selectnoise model outperform the random noise injection model @xcite ? (3) does the model's performance persist across different language families? and (4) does the unsupervised selectnoise model demonstrate competitive performance with supervised approach? based on these research questions, we have designed our experimental setup. ## results and discussions in this section, we will discuss results, observations and findings. the zero-shot automated evaluation scores are presented in tables 1 , 2 , 3 and 4. the results are reported with greedy, top k (@xmath1, and top-p (@xmath2.25) sampling strategies. these results indicate the importance of considering monolingual data from elrls in the modeling however, random noise injection and the select-noise approach outperform the obpe and bpe-dropout models, indicating the effectiveness of noise injection-based modeling techniques. in conclusion, the careful selection of noise candidates, as done in the selectnoise approach, has outperformed the random noise model (second best) and emerged as the state-of-the-art model. selective vs. random noise injection: unsupervised selective noise injection approaches exhibit a larger performance gain compared to the random noise injection model. this observation emphasizes the importance of a systematic selective candidate extraction and noise injection process. lexical vs. learned evaluation metrics: we observe a strong correlation between lexical matchbased metrics, such as bleu and chrf scores. further, semantic-based metrics like bleurt and comet exhibit similar trends to lexical match metrics, indicating a high level of correlation. this emphasizes the reliability of evaluation scores. automated vs. human evaluation: the proposed selectnoise model outperforms both baselines in human evaluation across all three languages. the model demonstrates acceptable zeroshot performance for elrls, with a strong correlation with automated evaluation scores. performance across language families: unsupervised selective noise injection consistently outperforms all the baselines across elrls, with few exceptions. the model exhibits similar performance trends across both language families. unsupervised vs. supervised noise injection: the unsupervised selectnoise model performs comparably to the supervised model, with slight variations depending on the language and family. the performance gap between the two models is minimal, indicating their equal strength. performance vs. sampling strategies: the performance with different sampling techniques is compared, and it is observed that the greedy approach for selectnoise performs better for the majority of languages. this finding indicates the existence of one-to-one lexical mapping across hrl and elrls. however, other sampling approaches are also effective for a subset of elrls. overall performance: as we can observe from the average automated evaluation scores, the proposed selectnoise model outperforms all the baselines by a significant margin. it also exhibits comparable performance to the supervised model, and this performance persists across different language families. these findings satisfy our hypothesis, leading us to conclude that the proposed se-lectnoise model is a state-of-the-art model for english-to-elrls mt systems. ## further analyses in this section, we perform a detailed analysis with selectnoise to understand factors contributing to performance gain and analyze robustness. performance trend with top-k and top-p: in figure 4 , the performance trend of the proposed model with varying values of k and p for top-p and top-k sampling is depicted. the candidate pool consists of a maximum of 61 characters (a range for k-value selection). the model performs best with a k-value of 50 and a p-value of 0.25, offering valuable insights for optimizing its performance through parameter selection. ## conclusion this study presents an effective unsupervised approach, selectnoise, for cross-lingual transfer from hrls to closely related elrls through systematic character noise injection. the approach involves extracting selective noise injection candidates using bpe merge operations and edit operations. furthermore, different sampling techniques are explored during the noise injection to ensure diverse candidate sampling. the model required only a small (1k example) monolingual data in elrls. the proposed model consistently outperformed strong baselines across 12 elrls from two diverse language families in the elrls-to-english mt task. the cumulative gain is 11.3% (chrf) over vanilla nmt. furthermore, the model demonstrated comparative performance to a supervised noise injection model. in the future, we will extend selectnoise to english-to-elrl mt task, as well as other nlg tasks and languages.