Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model
Anonymous Author(s)
AffiliationAddress email
Abstract
Numerous recent work on unsupervised machine translation (UMT) implies thatcompetent unsupervised translations of low-resource and unrelated languages,such as Nepali or Sinhala, are only possible if the model is trained in a massivemultilingual environment, where these low-resource languages are mixed withhigh-resource counterparts.
Nonetheless, while the high-resource languagesgreatly help kick-start the target low-resource translation tasks, the languagediscrepancy between them may hinder their further improvement.
In this work, wepropose a simple refinement procedure to disentangle languages from a pre-trainedmultilingual UMT model for it to focus on only the target low-resource task.
Ourmethod achieves the state of the art in the fully unsupervised translation tasks of
English to Nepali, Sinhala, Gujarati, Latvian, Estonian and Kazakh, with BLEUscore gains of 3.5, 3.5, 3.3, 4.1, 4.2, and 3.3, respectively.
Our codebase is availableat [-anonymous.4open.science/r/fairseq-py-BB
-]anonymous.4open.science/r/fairseq-py-BB44.
Introduction
Fully unsupervised machine translation (UMT), where absolutely only unlabeled monolingual corporaare used in the entire model training process, has gained significant traction in recent years.
Earlysuccess in UMT has been built on the foundation of cross-lingual initialization and iterative backtranslation [11, 13, 6, 26, 18].
While achieving outstanding performance on empirically high-resourcelanguages, such as German and French, these UMT methods fail to produce any meaningful translation(near zero BLEU scores) when adopted in low-resource cases, like the FLoRes Nepali and Sinhalaunsupervised translation tasks [10].
Arguably, such low-resource languages need UMT the most.
Liu et al.
[17] discovered that low-resource UMT models benefit significantly when they are initializedwith a multilingual generative model (mBART), which is pre-trained on massive monolingual corporafrom 25 languages (CC25).
Not only does this multilingual pre-training set contain data of thelow-resource languages of interest ( e.g., Nepali), it also comprises sentences from related languages( e.g., Hindi), which presumably boosts the performance of their low-resource siblings.
Tran et al.[27] and Nguyen et al.
[19] later propose ways to mine pseudo-parallel data from multiple directionsto improve low-resource UMT.
On the other hand, Conneau et al.
[7] find the curse of multilinguality ,which states that performance of a multilingual model with a fixed model capacity tends to declineafter a point as the number of languages in the training data increases.
Thus, adding more languagesmay hinder further improvement in low-resource UMT, which is intuitively due to incompatibilitybetween the linguistic structures of different languages.
Furthermore, these models have to telllanguages apart by using only a single-token language specifier prepended into the input sequence,which is shown to be insufficient [1].
Our analysis also shows that the models sometimes predictwords of the wrong language (see Appendix).
Meanwhile, as shown later in our analyses (§4.4), other complementary techniques such as pseudo-parallel data mining may have reached their limitsin improving low-resource UMT.
To alleviate the curse of multilinguality and work around the linguistic incompatibility issue in lowresource UMT, we propose a simple refinement strategy aimed at disentangling irrelevant languagesfrom a pre-trained multilingual unsupervised model, namely CRISS [27], and focusing exclusivelyon a number of low-resource UMT directions.
Briefly, our method consists of four stages.
In the firststage, we use a modified back-translation technique [24] to finetune an initial pre-trained multilingual
UMT model on English (En) and a family of low-resource languages
L with a different set offeed-forward layers in the decoder for each En ↔ L pair.
This step aims at discarding irrelevantlanguages and separating the languages L from each other.
The second stage separates the resulting
En ↔ L model into two En → L and L → En models.
This stage is motivated by the fact that
English is often vastly distinct from low-resource languages L , thus requiring a greater degree ofdisentanglement and less parameter sharing.
The third stage boosts the performance by using thesecond-stage models as fixed translators for the back-translation process.
The final stage, whichis only applied to from-English directions, is aimed to separate the target low-resource languagesbetween themselves into different models.
Overall, our method prioritises to maximize the individualperformance of each low-resource task, though the trade-off is that it can no longer translate multiplelanguages in a one-for-all manner.
In our experiments, our method establishes the state of the art in fully unsupervised translation tasksof English (En) to Nepali (Ne), Sinhala (Si), Gujarati (Gu), Latvian (Lv), Estonian (Et) and Kazakh(Kk) with BLEU scores of 9.0, 9.5, 17.5, 18.5, 21.0 and 10.0 respectively, and vice versa.
This isup to 4.5 BLEU improvement from the previous state of the art [27].
We also show that the methodoutperforms other related alternatives that attempt to achieve language disentanglement in vairouslow-resource unsupervised tasks.
Plus, our ablation analyses demonstrate the importance of differentstages of our method, especially the English disentanglement stage (stage 2).
Background
Recent advances in fully unsupervised machine translation (UMT) are often built on the foundationof iterative back-translation [24, 11, 3, 13].
It is a clever technique where the model back-translatesmonolingual data from one language to another, and the resulting pair is used to train the modelitself via standard back-propagation.
To make UMT successfully work, iterative back-translationmust be accompanied with some forms of cross- or multi-lingual initialization of the model, eitherthrough an unsupervised bilingual dictionary [12, 11], phrase-based statistical MT [13], or languagemodel pre-training [6, 26, 17].
Apart from that, cross-model back-translated distillation [18], wheretwo distinct models are used to complement the target model, can also boost UMT performance.
Plus, pseudo-parallel data mining, where sentences from two monolingual corpora are paired to formtraining samples through certain language-agnostic representations [29, 27, 19], has been shown tosignificantly improve UMT performance.
Nonetheless, when it comes to fully unsupervised translation of distant and low-resource languages[10], the aforementioned techniques fail to translate their success in high-resource languages tolow-resource counterparts, unless “multilinguality” is involved.
mBART [17] is among the firstmultilingual model to boost the performance greatly in low-resource UMT, by pre-training the modelon the CC25 dataset, which is a 1.4-terabyte collection of monolingual data from 25 languages.
The
CC25 dataset contains not only data from English and the low-resource languages of interest, like
Nepali, but also data from their related but higher-resource siblings, like Hindi.
CRISS [27] later advances the state of the art in low-resource UMT by finetuning mBART withpseudo-parallel data of more than 180 directions mined from the model’s encoder representations.
This type of UMT models handles translations in multiple directions by a few principles that aredesigned to maximize parameter sharing.
First, it builds a large shared vocabulary that coverswordpieces from all languages.
Second, the encoder encodes input sentences without knowledgeabout their corresponding languages to promote language-agnostic representations in its outputs.
Lastly, the decoder receives the encoder outputs and decodes the translation in the target language byprepending a language-specific token at the beginning of the output sequence.
Training the modelin such a multilingual environment helps the encoder learn language-agnostic latent representationsthat are shared across multiple languages, allowing the decoder to translate from any language.
The vast availability of high-resource siblings ( e.g., Hindi for Indic languages) may help improvethe performance of low-resource MT significantly [9, 5].
Meanwhile, LAgSwAV [19] improveunsupervised MT by mining pseudo-parallel data from monolingual data with cluster assignments.
Despite its outstanding success, multilinguality may conversely hinder the model’s furtherimprovement in low-resource unsupervised translation due to the curse of multilinguality [1, 7].
A multilingual model with a fixed capacity may perform suboptimally on individual languages asit is forced to simultaneously handle conflicting structural reorderings of distant languages [16].
The single-token language specifier is also not enough to ensure the language consistency of itspredictions [1].
Our proposed method aims to gradually disentangle the languages and focus onthe target low-resource directions only.
One prominent feature of our method is that it prioritises todisentangle English from the remaining low-resource languages first, as these languages are oftenmuch more distant from the common language English than from themselves [28].
There are several efforts with similar objectives as ours, albeit of different contexts and scenarios.
Specifically, Sen et al.
[23] propose a shared encoder and language-specific decoder model to betrained with back-translation for unsupervised MT, which is slightly similar to stage 1 of our proposedfour-stage refinement process, as explained in §3.
Our stages 2, 3 and 4 improve the first stage,or equivalently Sen et al.
[23], further by refining the model in ways that are not possible in stage1.
Meanwhile, Sachan and Neubig [22], Zhang et al.
[30], Li and Gong [16] propose methods tocause the model to implicitly “select” relevant language-specific parameters by gating mechanisms orgradient readjustments, in the context of supervised multilingual translation.
However, these methods,as found in §4, struggle with unsupervised MT as signals from back-translated synthetic trainingsamples are often much noisier.
On the implementation side, the design of our language-specific FFNlayers (§3.1) is also largely inspired by sparsely-distributed mixture of experts [25, 14, 31].
Low-resource MT can also benefit greatly from the presence of supervised parallel data, or auxiliarydata, in a related language ( e.g., English-Hindi for English-Nepali), which is not a fully unsupervisedsetup but a zero-shot scenario.
Garcia et al.
[8, 9] propose multi-stage training procedures thatcombine both monolingual data and auxiliary parallel data to improve the performance of low-resourcetranslation.
Meanwhile, Chen et al.
[4, 5] suggest to scale their experiments up to 100 languages toattain better performance boosts.
Nonetheless, while such line of work offers considerable insightinto low-resource MT, it is not within the scope of this paper, as we focus on fully unsupervisedmachine translation where absolutely no parallel data is available, regardless of languages.
Language disentanglement refinement
In this section, we describe our multilingual language disentanglement refinement procedure thatconsists of four stages.
Before the training process starts, we first choose to restrict the model, e.g.,
CRISS [27], to only translate English (En) to (and from) a small group L = ( l 1 , l 2 , ..., l N ) of Nlow-resource languages based on their genealogical, demographic, areal or linguistic similarities,such as Nepali (Ne), Sinhala (Si) and Gujarati (Gu) for Indic family (a part of Indo-European).
Thereis no guarantee, however, that these languages are indeed significantly close due to such heuristics.
Inthe following, we first propose a language-specific feed-forward layer (§3.1), which helps achieve agradual language disentanglement process (Figure 1).
Then, in §3.2-§3.5, we explain the four stagesof our language disentanglement procedure, which is also visually demonstrated in Figure 2.
3.1 Language-specific sharded feed-forward layer
While our method starts with finetuning a fully parameter-shared multilingual MT model, we desire togradually and progressively increase the level of language disentanglement by reducing the degree ofparameter sharing of the model.
Thus, for every layer j of an H -layer Transformer decoder such thatj is divisible by a constant σ ( e.g., 3) and 1 ≤ j ≤ H , we propose to replace its vanilla feed-forwardlayer (FFN j ) with N separate layers FFN j,i such that each corresponds to a language l i ∈ L .
During the training process, each of the separate FFN j,i layers is sharded and reside individuallyon a distinct GPU- i accelerator, where their back-propagated gradients are not aggregated oraveraged during the data parallelism process.
Each GPU- i device is only fed with data from itsrespective language l
i and the common language English.
Meanwhile, the remaining parameters( e.g., embedding and attention layers) are still shared and their gradients are averaged across all GPU devices during back-propagation.
Figure 1 visualizes a sharded FFN layer of our model and how itsparameters are allocated on each GPU.
During inference, FFN j,i is used to decode the translation toand from l i .
Formally, let ˆ θ = { ˆ θ m , ˆ θ f } be a fully parameter-shared multilingual model ( e.g., CRISS) at ourinitialization, where ˆ θ m is the set of parameters intended to be shared and ˆ θ f is the initial setof FFN parameters intended to be disentangled in our proposed model.
We define our model asθ =
{ θ m , θ 1 f , θ 2 f , ..., θ Nf } , where θ m denotes the shared parameters while θ if denotes the separateparameters of all FFN j,i layers ( 1 ≤ j ≤ H ) for a language l i ∈ L .
Before we begin training themodel, we initialize θ m = ˆ θ m and θ if = ˆ θ f , ∀ i ∈ [1 , ..., N ] .
Then during each update step, thegradients of θ if and θ m are computed as follow:
[Equation]
[Equation]
[Equation]
where X i and X e are monolingual corpora (or data distribution) of language l i and English respectively,
J θ ( x | y )
= − log P θ ( x | y ) , and y ei ∼ P ( ·| x i , { θ m , θ if } ) is the back-translation into English from x iwhile y ie ∼ P ( ·| x e , { θ m , θ if } ) is the back-translation into l i from English by the model { θ m , θ if } .
Equations 2 and 3 show that gradients are not aggregated for the language-specific sharded FFNlayers, while the remaining are aggregated and averaged across all data streams.
However, note thatour model only separates the FFN layers of the decoder, while all encoder parameters are fully sharedto ensure the language-agnostic representations from the encoder [27, 19].
Implementation aspect.
The rationale behind the design of our language-specific sharded FFNs isto take full advantage of multi-GPU environment to maximize training efficiency both in speed andbatch size.
First, each sharded FFN is placed on a separate GPU and only the respective languagespecific data streams are fed into that GPU, which enables us to achieve the same maximal batch sizeas the regular Transformer without running into out-of-memory error.
Second, FFN layers, ratherthan other types of layers ( e.g., attention), are chosen to be separated because they require minimaladditional memory (only 50Mb for a 8.8Gb model) to achieve noticeable performance gain (see§4.2).
If we have access to only one GPU, we can allocate all FFNs into it, reduce the batch size andincrease gradient accumulation to achieve the same multi-GPU effect.
3.2 First stage
[Figure]
With the aforementioned language-specific FFN layers, we enter the first stage of refinement.
Thepurpose of this stage is to begin differentiating the language-specific FFN layers so that each FFN canspecialize in its designated language.
That is, after initializing our model θ = { θ m , θ 1 f , ..., θ Nf } withthe pre-trained multilingual UMT model ˆ θ = { ˆ θ m , ˆ θ f } as θ m =
ˆ θ m and θ if = ˆ θ f ∀ i ∈
[1 , ..., N ] , wefinetune it with iterative back-translation [6], except with some noteworthy modifications.
Specifically,we uniformly sample monolingual sentences x e and x i from English and each low-resource languagel i ∈ L , respectively.
Then, we use θ i
= { θ m , θ if } , a subset of θ , to back-translate x e and x i intoy ie and y ei , respectively.
After that, we train the model with y ie → x e and y ei → x i pairs.
Note thatfor each direction of back-translation, forward and backward directions, the appropriate FFN layersθ if are used with respect to the data stream involving language l i and English on a separate GPUi .
Furthermore, similarly to Conneau and Lample [6], the back-translation model is continuouslyupdated after each training step.
The resulting model θ (1) =
{ θ (1) m , θ 1 , (1) f , ..., θ N, (1) f } will be used toinitialize the second stage of refinement.
3.3 Second stage
Alleviating the curse of multilinguality may involve disentanglement of unrelated languages.
Yet, formany low-resource translation tasks, English is often significantly distant from the target low-resourcecounterparts, such as Nepali and Sinhala, which in fact share certain similarities between themselves.
Thus, English may not share similar structural patterns with any of the target languages [28].
Despitethat, most existing UMT models are bidirectional [6, 26, 18].
This means that they have to handle bothtranslations to English and their target low-resource language l i , causing them to endure reorderingdiscrepancy and perform sub-optimally for both directions.
The second stage is aimed to disentangle English .
It builds two separate models θ (2) →L and θ (2)
→ Enfrom θ (1) , which are specialised to translate to target low-resource languages l i ∈ L and to English,respectively.
Specifically, the second stage starts by initializing both θ (2) →L
and θ (2) →
En with θ (1) .
Wethen finetune them with iterative back-translation exclusively in En → l i and l i → En directions,respectively.
Different from the first stage, we use a fixed model θ (1) to back-translate the monolingualdata.
In other words, to finetune θ (2) →L , we first initialize it with θ (1) .
Then we sample monolingualdata x i uniformly from all l i ∈ L and use the fixed model θ (1) to back-translate them into Englishy ei ∼ P ( ·| x i , { θ (1) m , θ i, (1) f } ) .
The resulting pair y ei → x i is used to train θ (2) →L .
In the oppositedirection, we finetune θ (2) →
En by first initializing it with θ (1) , then sample English data x e andrandomly choose an l i ∈ L to back-translate x e into y ie ∼ P ( ·| x e , { θ (1) m , θ i, (1) f } ) , and finally trainθ (2)
→ En with y ie → x e pairs.
3.4 Third stage
The third stage improves the performance further by continuing finetuning θ (2) →L and θ (2)
→
En into θ (3) →Land θ (3) →
En , respectively.
Specifically, we build θ (3) →L by initializing it with θ (2) →L and use θ (2) →
En as afixed back-translation model and train θ (3) →L in the same manner as in §3.3.
Similarly, θ (3) → En is trainedin the same fashion in the opposite direction.
3.5 Fourth stage
The last stage is aimed to achieve the maximal degree of language disentanglement not only from
English, but also between all languages in the target group.
Specifically, we seek to train Nseparate models θ (4)
→ l i for each language l i ∈ L .
Each model θ (4) → l i is initialized with θ (3) →L withthe corresponding language-specific FFN layers, i.e., { θ (3) m , θ i, (3) f } →L , and θ (4) → l i is trained by backtranslation with a fixed backward model θ (3)
→ En , and exclusively on En → l i direction.
Because onlyone direction is trained in this stage, we no longer use the above language-specific FFN layers forour end models.
Instead, we revert back to the vanilla Transformer, whose FFN layers are initializedwith the appropriate language-specific layers from the previous stage.
Note that we do not performthis stage of refinement for the to-English ( → En ) direction, due to the fact that the encoder isfully shared across all languages and there is no difference in separating l i → En directions for theencoder, while the decoder only decodes the common English language.
Experiments
In this section, we tested our method on the standard low-resource unsupervised machine translation(§4.1).
We then empirically analyze various aspects of our method and compare it with other possiblevariants (§4.2) as well as other related alternatives in the literature (§4.3).
Low-resource unsupervised translation
Setup.
We evaluate our method on the FLoRes [10] low-resource unsupervised translation tasks on
English (En) to and from Nepali (Ne), Sinhala (Si) and Gujarati (Gu) from the Indic family, as wellas other low-resource languages: Latvian (Lv) and Estonian (Et) from the Uralic family and Kazakh(Kk) from the Turkic family.
Although Hindi (Hi) and Finnish (Fi) are relatively high-resourcedcompared to their respective siblings, we still add them into the mix for Indic and Uralic familiesrespectively to help the learning process of their respective low-resource siblings, following [9, 5].
We start finetuning our model using the exact same codebase and the published pre-trained CRISSmodel [27].
We also use the same sentencepiece pre-processing and vocabulary.
Specifically, wegroup the tested languages into 3 groups: Indic (En-{Ne,Si,Hi,Gu}), Uralic (En-{Fi,Lv,Et}) and
Turkic (En-Kk).
For each group, we use monolingual corpora from the relevant languages of the
Common Crawl dataset, which contains data from totally 25 languages [17].
We limit the amountof monolingual data per language to up to 100M sentences.
Similarly to CRISS, our model is a12-layer Transformer, whose decoder’s FFN layers are replaced with our language-specific sharded
FFN layers with σ = 3 (see §3.1) during the first 3 stages.
For each group of N languages, we use
N GPUs to train the model and shard each language-specific FFN layer to a single GPU, where wechannel only data streams of the respective languages, according to our formulations in §3.2.
We usea batch size of 1024 tokens with a gradient accumulation factor of 2 [20].
Similarly to CRISS, weuse the multi-bleu.perl script to evaluate the BLEU scores with a beam size of 5.
In each stage, wefinetune the models for 20K updates.
Further details are provided in the Appendix.
Results.
Table 1 shows the performances of our method against recent best baselines, namelymBART [17], LAgSwAV [19] and CRISS [27], on the fully unsupervised translation tasks between
English and the low-resource languages from the Indic, Uralic and Turkic families.
As shown, ourmethod achieves the state of the art in En-Ne and En-Si FLoRes low-resource UMT tasks, with 9.0and 9.5 BLEU, as well as other languages across the three families, such as 21.0 and 10.0 BLEU for
En-Et and En-Kk tasks.
It surpasses the previous best [27] by up to 5.8 BLEU and achieves a meangain of 2.6 BLEU across the 16 directions.
The results also indicate that our method achieves moregain margin for low-resource languages (Ne, Si, Gu) compared to their high-resource siblings (Hi).
[Table]
[Table]
4.2 Ablation study
In this section, we conduct a comprehensive ablation study to gain more insights of our method andthe importance of its components.
In particular, we examine the contribution of each stage of ourprocedure to the overall performance gain across various low-resource languages (Table 2).
We alsoinvestigate how our decoder language-specific FFN layers perform in comparison to other possiblevariants, such as language-specific attention layers.
Stage-wise ablation study.
Table 2 shows the performances at different refinement stages of ourmethod in low-resource fully unsupervised translation between English to and from Nepali, Sinhala,
Gujarati, Latvian, Estonian and Kazakh.
As it can be seen, our first and second stages achievesignificant kick-off gain from the CRISS baseline, with up to 5.1 BLEU score for Gu-En direction.
We also observe considerable improvement from stage 1 to stage 2, with up to 2.7 BLEU.
Thisindicates that separation of English from other low-resource languages play a crucial role.
Meanwhile,the third and fourth stages improve the performances further, though with less margin.
In addition, itis noteworthy that stage 4 does not noticeably outperforms stage 3 for to-English ( → En ) direction,due to the fact that the encoder is fully shared across all low-resource languages and there is nodifference in disentangling l i → En directions from the encoder’s perspective.
Thus, there is nopoint training stage 4 for to-English direction, as formulated in our stage 4 (see §3.5).
Language-specific variants.
Other than our decoder language-specific FFN layers explained in§3.1, there could be other possible ways to shard model parameters into language-specific components.
We compare some of these variants against our own in Table 3 to examine their effects on the FLoReslow-resource unsupervised translation tasks with Nepali, Sinhala and Gujarati.
Specifically, wecompare our method (Lg.
FFN/Dec) with a similar variant without the language-specific FFN layers(w/o
Lg.
FFN/Dec), with decoder language-specific self- and cross-attention layers (Lg.
Att/Dec),with language-specific FFN layers in the encoder (Lg.
FFN/Enc), and finally with language-specific
FFN layers in both encoder and decoder (Lg.
FFN/Enc+Dec).
To ensure consistency, we only showresults up to stage 3 of our procedure and skip the fourth stage, where language-specific elementsno longer apply.
In addition to the BLEU scores, we also report the percentage of excess memoryoverhead needed for each method compared to the baseline CRISS (0% overhead).
As shown in Table 3, the variant without any language-specific parameters (w/o Lg.
FFN/dec)underperforms our method by up to 1.0 BLEU, even though English is separated in stage 2 and stage3.
This indicates that it is essential for the method to have a gradual language disentanglement process.
Furthermore, placing the language-specific FFN layers on the encoder only (Lg.
FFN/Enc) causeseven more detrimental performance drop.
The results imply that it is not advisable to linguisticallyseparate the encoder only.
Similarly, having language-specific parameters on the attention layers
[Table]
(Lg.
Att/Dec) does not perform as good as our FFN counterpart, despite having the same memoryfootprint overhead of 4%.
Meanwhile, placing language-specific FFN layers on both encoder anddecoder (Lg.
FFN/Enc+Dec) yields relatively similar results as our own method, although it requiresdouble the memory overhead.
Thus, our choice of separate FFN parameters on the decoder only ismore favorable given its outperforming capability while consuming less memory.
The Appendixprovides further ablation analysis this issue.
4.3 Comparison with related methods
In this section, we compare our refinement procedure against other related methods, as well as thosethat share a similar language disentanglement objective as ours.
Particularly, we reproduce the methodproposed by Sen et al.
[23] in low-resource unsupervised MT.
This method is equivalent to stage 1 ofour refinement procedure, except that the entire decoder is linguistically disentangled with separatedecoders for different languages, not just FFN parameters.
Meanwhile, Sachan and Neubig [22] and
Zhang et al.
[30] both share the same goal of alleviating the curse of multilinguality, but in the contextof supervised multilingual translation, where the model teaches itself to implicitly “select” relevantlanguage-specific layers by gating mechanisms or gradient readjustments.
We adapt these methods tounsupervised MT by substituting supervised training data with multilingual back-translated samplesby the models.
Lastly, Zuo et al.
[31] recently propose a mixture-of-experts (MoE) variant [14, 25],where experts (FFNs) are selected randomly to perform the forward pass.
In the context of languagedisentanglement, these separate experts act as implicit language-specific components.
We also adaptthis method to low-resource unsupervised tasks with multilingual back-translation and set the numberof experts as the number of languages.
For all methods, we use CRISS as the initial model.
The comparison results are shown in Table 4.
As shown, the method proposed by Sen et al.
[23]performs slightly above stage 1 of our method, while it underperforms our complete procedure byup to 2.7 BLEU in Gu-En task.
This is in line with the fact that it is almost equivalent to our firststage.
The approaches suggested by Sachan and Neubig [22] and Zhang et al.
[30], on the other hand,yield considerably lower BLEU scores in various low-resource unsupervised tasks.
A possible reasonfor this underperformance is that these methods are designed for supervised multilingual translation,where accurate parallel data plays a crucial role.
When adapting to the unsupervised regime, thesynthetic training samples created by back-translation may be too noisy and inaccurate.
Meanwhile,the MoE candidate [31] also produce lower BLEU scores than our method, which can be due to itslack of explicit language-specific components to enforce a language disentanglement naturally.
4.4 Limit of other unsupervised techniques
Apart from iterative back-translation, there are other techniques that have been shown to improveunsupervised machine translation.
One of them is cross-model back-translated distillation [18], or
CBD, where two distinct UMT teachers are used to distill the final model.
Another is pseudo-paralleldata mining [27, 19], where language-agnostic representations are built for sentences across differentlanguages, which are then used to map unlabeled sentences from one language to another to createsynthetic parallel data [2].
CRISS is also itself pseudo-parallel data mining method.
In this segment,we demonstrate that the aforementioned techniques do not adapt well with low-resource unsupervisedtasks, which shifts our attention to other areas, like the curse of multilinguality. Specifically, we apply
[Table] [Table]
[Table]
CBD on CRISS by finetuning two distinct models from pre-trained CRISS and using them to distill afinal model.
For pseudo-parallel mining, we use LASER [2] to mine pseudo-parallel data in only thelow-resource directions of interest using the encoder outputs from CRISS itself.
The results are reported in Table 5.
As shown, CBD only performs relatively similar to our stage 1.
This indicates that the CBD only improves due to back-translation, and not its own distillation effect.
Nonetheless, CBD faces a disadvantage that the two teachers are not considerably distinct as they areboth finetuned from the same pre-trained CRISS model, which is not advisable in the original paper.
This is nonideal as there is no existing well-performing pre-trained model other than CRISS.
Meanwhile, the mined pseudo-parallel data contributes little to the performance improvement dueto the reality that the amount and quality of mined data is too low for low-resource pairs, which isless than 5000 pairs for En-Ne and En-Si.
This forces the model to be trained on such small mineddatasets by progressively decreasing loss weight to prevent overfitting.
While many may attributethis to the model’s failure in mining more high-quality pseudo-parallel data, we empirically showthat it is more likely that there are in fact not enough real parallel data in low-resource corpora to bemined!
Specifically, we use our outperfoming model to back-translate the entire English monolingualcorpus to Nepali.
For each resulting back-translated Ne sentence, we search for its closest real Nesentence by token-based Levenshtein distance [15] in the Ne monolingual corpus, and then filter outpairs whose distance is more than 20% the sentence length.
We repeat the process for En-Si as well.
As shown in Table 6, out of >100M possible unfiltered pairs, only <6000 samples satisfy the filteringcriterion.
We provide an in-depth analysis on this issue in the Appendix.
Conclusion
We proposed a simple refinement procedure to alleviate the curse of multilinguality in low-resourceunsupervised machine translation.
Our method achieves the state of the art in the FLoRes low-resourcetask of English-Nepali and English-Sinhala, as well as other language low-resource pairs across 10language directions in Indic, Uralic and Turkic families.
Further analyses show that the methodoutperforms other related alternatives when adapting to low-resource unsupervised translation