Improving NMT from a Low-Resource Source Language: A Use Case from C atalan to C hinese via S panish (2024)

Abstract: The effectiveness of neural machine translation is markedly constrained in low-resource scenarios, where the scarcity of parallel data hampers the development of robust models. This paper focuses on the scenario where the source language is low-resourceand there exists a related high-resource language, for which we introduce a novel approach that combines pivot translation and multilingual training. As a use case we tackle the automatic translation from Catalan to Chinese, using Spanish as an additional language. Our evaluation, conducted on the FLORES-200 benchmark, compares our new approach against a vanilla baseline alongside other models representing various low-resource techniques in the Catalan-to-Chinese context. Experimental results highlight the efficacy of our proposed method, which outperforms existing models, notably demonstrating significant improvements both in translation quality and in lexical diversity.

## introduction the development of neural machine translation (nmt) has considerably benefited translation between language pairs abundant in parallel data, enhancing translation accuracy and fluency across diverse linguistic landscapes @xcite . however, its effect is challenged by the fact that building an effective nmt system requires a large amount of parallel data. this challenge is particularly pronounced in the case of low-resource languages, that is, language pairs with limited parallel language resources, remaining a significant hurdle in achieving universal communication. an exemplary case highlighting such hurdle involves the translation dynamics between catalan and chinese, (ca-zh) two languages characterized by limited parallel corpora. the year 2022 marked a significant increase in chinese investments in catalonia foot_0 , and the chinese population emerged as the fourth largest foreign community in catalonia foot_1 . these together highlight the growing economic and social interactions between these regions and thus the pressing need for effective communication tools between catalan and chinese speakers. despite the potential benefits, the development of a robust nmt system for the ca-zh language pair faces notable challenges, primarily due to the scarcity of direct parallel data. addressing this gap, previous works have sought to navigate the low-resource landscape of the ca-zh language pair. the research by @xcite was the first work to specifically focus on addressing the low-resource ca-zh language pair, where they broke new ground by generating non-human-written parallel sentences, i.e. pseudo-parallel corpus via pivot translation and then used them to train @xmath0. another work @xcite concerned building ca-zh parallel data, where ca-zh bitexts was first mined from wikipedia with the help of the open-source laser tookit 3 and then passed to san-ity check according to @xcite methodology. therefore, unlike the training datasets created by @xcite , @xcite parallel corpus consists of human-selected bitexts. subsequently, @xcite @xcite dataset to fine-tune a massively pre-trained multilingual nmt model, i.e. m2m-100 with parameters of 418m by @xcite for ca↔zh, presenting better translation performance in both directions as compared to the original m2m-100. furthermore, other work @xcite @xcite , related to building parallel data for many language pairs, included ca-zh. the multilingual bitexts therein were massively mined from web-based resources and subsequently utilized to train multilingual nmt models like m2m-100, which is also workable with the ca↔zh translation. the previous studies on the ca-zh language pair contributed to enhancing the automated translation between these two languages. however, each focused primarily on employing a singular, specific low-resource nmt technique, e.g. pivot translation in @xcite , finetuning in @xcite , multilingual training in @xcite , etc. unlike these approaches, our work aims to propose a novel integration, pivot translation-aided multilingual training (ptamt), and compare it against existing methods (multilingual training, fine-tuning, pivot translation, finetuning coupled with pivot translation). we focus on the @xmath1, as a use case in which the source language is low-resource and there is a related higher-resourced language, spanish (es). the technique we introduce, ptamt, uses additional data from es both as pivot and as multilingual training. the contributions of our work can be summarized as follows: ## proposed method our proposed method, ptamt, couples pivot translation with multilingual training to leverage the advantages of both techniques. distinct from previous work @xcite , which uses a less related pivot language to initially train a multilingual nmt system for back-translating and synthesizing pseudoparallel data, ptamt employs a pivot language (es) that is linguistically closer to the source language. this choice is informed by the synthetic pivot translation approach demonstrated by @xcite , which is favored due to the greater linguistic affinity between the source (ca) and pivot (es) languages as compared to the pivot (es) and target (zh) languages @xcite . in our implementation, we used an existing es-zh corpus to synthesize pseudo-parallel ca-zh data by translating es sentences into ca. this strategy aligns the synthetic side with the source language (ca) and uses authentic data for the target language (zh), enhancing the model's ability to produce natural output. nevertheless, as pivot-translated sentences are inherently machine-translated texts, they are prone to containing machine translationese. to address this, ptamt strategically leverages the es-zh bitexts in a multilingual training setup to facilitate effective pivot-based knowledge transfer. this approach helps to mitigate the potential impact of noise introduced by the synthetic ca input. ptamt incorporates both source languages (ca and es) in the encoder, and applies encoder parameter sharing throughout training, which is applicable for both from-scratch training and finetuning. the same set of weights and biases is shareable in a single encoder for inputs from both source languages. multilingual training empowers the model to capture and integrate contextual information from both source languages. given that the es data is authentic (human-produced), we hypothesize that ptamt will aid in reducing the influence of noise from the pivot-translated ca training data on the target zh output. ## conclusions and future work in this work, our comprehensive experimental evaluation of from-scratch trained and m2m-100-418m pre-trained models for the @xmath2. significantly, these experiments have confirmed the substantial benefits of these methods on translation quality and lexical diversity, with our novel ptamt method emerging as a key innovator in addressing the challenges inherent in translating the low-resource language pair. the ptamt method, with its ability to effectively utilize pseudo-parallel and authentic parallel data, significantly mitigates the influence of source-side machine translationese and enhances the model's capability to produce translations that are not only accurate but also linguistically rich and varied. this approach not only broadens the lexical range and usage contexts available to the model but also ensures a nuanced understanding and replication of natural language subtleties, as evidenced by the improved lexical diversity metrics and the accurate handling of complex linguistic phenomena such as negative concord. moreover, our findings seem to imply the critical role of integrating authentic data in the desired language pair(s) into the training or fine-tuning process, demonstrating that even a small amount of authentic parallel data can substantially elevate the performance of nmt systems. this insight emphasizes the importance of combining pseudoparallel and authentic inputs to achieve the best possible translation outcomes, particularly in the context of low-resource language pairs. while our study marks progress in nmt for the low-resource ca-zh pair, it also unveils areas which deserve further exploration. the potential domain alignment between our test set flores-200 and the original ca-zh-wiki parallel data raises questions about how the inclusion of a modest amount of authentic parallel data in the target language pair(s) influences translation outcomes. this becomes especially relevant when considering the potential for domain-specific biases to affect the evaluation of nmt systems. moreover, the observed discrepancies in how from-scratch trained ptamt-enhanced models and m2m-100-418m ptamt-enhanced models handle linguistic complexities such as negative concord-both before and after additional fine-tuning-suggest underlying differences in model learning dynamics that deserve closer scrutiny. these divergent model responses highlight the need for a nuanced understanding of how different training approaches impact the nmt models' ability to grasp and accurately render complex linguistic structures. to navigate these uncertainties and expand upon our findings, we propose several avenues for future research: firstly, building novel and diversified test sets to quantify and generalize the influence of authentic parallel data in the target language pair(s) on model performance; secondly, exploring the models' internal representations and additional fine-tuning processes to pinpoint factors contributing to their distinct responses to linguistic complexities such as negative concord; thirdly, expanding our investigation to include more low-resource language pairs to enable a comprehensive evaluation of the ptamt method's applicability across diverse linguistic contexts.