C har S pan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languages (2024)

Abstract: We address the task of machine translation (MT) from extremely low-resource language (ELRL) to English by leveraging cross-lingual transfer from *closely-related* high-resource language (HRL). The development of an MT system for ELRL is challenging because these languages typically lack parallel corpora and monolingual corpora, and their representations are absent from large multilingual language models. Many ELRLs share lexical similarities with some HRLs, which presents a novel modeling opportunity. However, existing subword-based neural MT models do not explicitly harness this lexical similarity, as they only implicitly align HRL and ELRL latent embedding space. To overcome this limitation, we propose a novel, CharSpan, approach based on character-span noise augmentation into the training data of HRL. This serves as a regularization technique, making the model more robust to lexical divergences between the HRL and ELRL, thus facilitating effective cross-lingual transfer. Our method significantly outperformed strong baselines in zero-shot settings on closely related HRL and ELRL pairs from three diverse language families, emerging as the state-of-the-art model for ELRLs.

## introduction recent advancements in multilingual modeling have expanded the coverage of natural language processing (nlp) technologies to many lrls by transferring knowledge from hrls to lrls. as a result, this progress has led to remarkable advancement in multiple nlp tasks, including mt, transliteration, natural language understanding, and text generation @xcite @xcite for lrls. however, most of the existing work has focused on the top few hundred languages * work done during first author's internship at microsoft. he was enrolled as a graduate student at iit hyderabad at that time. ## related work traditionally, character-level noise has been used to improve the robustness of mt systems to spelling mistakes and asr errors @xcite @xcite . however, these approaches are mostly investigated for their impact on robustness rather than for cross-lingual transfer. more recently, token/bpe-level general noise augmentation approaches such as worddropout @xcite and switchout @xcite have been proposed, but they have limited cross-lingual transfer capabilities. close to our work, @xcite and @xcite show that augmenting data with character-level noise can help cross-lingual transfer. the models were evaluated with nlu tasks. n contrast, our work focuses on mt, an nlg task, which is much more challenging than an nlu task in a zero-shot setting. furthermore, we explore span noise augmentation, which considers larger lexical divergence (less lexical similarity between the hrl and elrl) and enables better cross-lingual transfer. in other work on utilizing lexical similarity, patil et al. (2022) proposed overlapbpe, which takes lexical overlap between hrl and lrl into account while learning bpe vocabulary. provilkov et al. (2020) introduced bpe-dropout, providing on-the-fly non-deterministic segmentations while training. soft decoupled encoding (sde) wang et al. (2019) utilizes lexical information without pre-segmenting the data by decoupling the lexical and semantic representations. sde requires small monolingual data for modeling. in contrast, the charspan model does not require any training resources for elrls. it only needs script similarity between the hrl and elrl. ## experimental setup we seek answers to the following questions: ## results and analyses the proposed charspan and baseline models' results across different language families are presented in noise vs. baselines: all the proposed noise augmentation models outperform vanilla nmt and all baseline models that utilize lexical similarity (i.e., obpe, bpe-dropout, and sde). this trend is consistent across all language families and elrls. moreover, existing lexical similarity-based baselines do not provide any major improvement in translation quality over vanilla nmt. possible reasons for this can be twofold: (1) most of the elrls either do not have monolingual data (obpe and sde are required) or have small data, and (2) we observe that in obpe, approximately 90% of vocabulary tokens are already overlapping among hrls and elrls, leaving little room for learning additional overlapping tokens. this is expected, as these two language sets are closely related. the proposed charspan method also outperforms general data augmentation methods like (sub)worddropout and (sub)wordswitchout, showing its effectiveness. ## conclusion this study presents a simple yet effective novel character-span noise argumentation model, charspan, to facilitate better cross-lingual transfer from hrls to closely related elrls. the approach generalizes to closely related hrl-elrl pairs from three typologically diverse language families. the proposed model consistently outperformed all the baselines. to the best of our knowledge, we are the first to apply noise augmentation for the nlg task. in the future, we will extend charspan to other nlp tasks, combine it with pre-trained models, and investigate noise augmentation in english-to-elrl mt task.