C ONTRASTIVE C LUSTERING TO M INE P SEUDO P ARALLEL D ATA FOR U NSUPERVISED T RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iterative backtranslation.
However, there may exist small amount of actual parallel data hidden in the sea of unlabeled data, which has not been exploited.
We develop a new ﬁne-tuning objective, called Language-Agnostic Constraint for SwAV loss, or
LAgSwAV, which enables a pre-trained model to extract such pseudo-parallel data from the monolingual corpora in a fully unsupervised manner.
We then propose an effective strategy to utilize the obtained synthetic data to augment unsupervised machine translation.
Our method achieves the state of the art in the WMT’
English-French, WMT’16 German-English and English-Romanian bilingual unsupervised translation tasks, with 40.2, 36.8, and 37.0 BLEU, respectively.
We also achieve substantial improvements in the FLoRes low-resource English-Nepali and English-Sinhala unsupervised tasks with 5.3 and 5.4 BLEU, respectively.
I NTRODUCTION
The quest to build a fully unsupervised machine translation (UMT) system, where only unlabeled monolingual corpora are available, has received increasing attention in recent years.
Profoundly,
Lample et al.
(2018a;c) introduced the general principles for UMT that include cross-lingual initialization, language modeling and iterative back-translation.
Although various UMT variants (Lample et al., 2018a;c; Conneau & Lample, 2019; Song et al., 2019; Liu et al., 2020; Nguyen et al., 2021) applied these principles differently, they ultimately train their models on noisy synthetic parallel data generated by the models themselves or some randomization processes, which may cause harm as the generated synthetic data is often noisy and low-quality, especially at the beginning of training.
However, these methods may have missed out a potential that some parallel (or comparable) data may exist in the monolingual corpora, which can be effectively mined to augment the UMT training.
This paper is motivated to explore mining of pseudo-parallel data for UMT tasks.
While there have been limited research in unsupervised mining of such data (Wu et al., 2019a;b), there have been several studies on bitext mining from sentence embeddings in semi-supervised translation or zeroshot transfer setups (Zweigenbaum et al., 2018; Schwenk, 2018; Artetxe & Schwenk, 2019a;b).
However, these methods require the models to be pre-trained on massive multilingual parallel data, which renders them inapplicable and incomparable to the fully unsupervised setup.
Furthermore, such models may not behave well off-the-shell on low-resource languages that were not in the pretraining data, such as Nepali and Sinhala (Guzm´an et al., 2019).
Meanwhile, applying these mining algorithms directly to sentence embeddings produced by fully self-supervised models (Conneau &
Lample, 2019) leads to a signiﬁcant amount of misalignments, which hurts the UMT models as shown later in our experiments (§4).
In this paper, we propose a novel modiﬁcation to SwAV loss (Caron et al., 2020), called LanguageAgnostic Constraint for SwAV ( LAgSwAV ) 1 , which is used to ﬁne-tune existing pre-trained selfsupervised models (Conneau & Lample, 2019; Liu et al., 2020), so that they become semantically contrastive and language-agnostic .
Speciﬁcally, semantic contrastiveness is deﬁned, in this paper, as the model’s ability to cluster sentences of similar semantic contents into similar groups.
Meanwhile,
[Figure]
language-agnostic property means that the embedding space of sentences from different languages are indistinguishably mixed together.
Figure 1 further illustrates the two properties.
We argue that these two properties combined can help the model mine more accurate pseudo-parallel data, since language-agnosticism reduces the distances between cross-lingual embeddings while semantic contrastiveness helps bring sentences of similar contents closer together even if they are in different languages .
To achieve the semantically contrastive property, we adopt the SwAV loss (Caron et al., 2020), which is designed to classify input data into different clusters that reﬂect certain semantic characteristics.
However, since SwAV is not designed to be language-agnostic, we introduce a new language-agnostic constraint for it to enable the latter property.
Analysis experiments in §4 show that our constraint signiﬁcantly outperforms vanilla SwAV loss as well as other baselines in mining pseudo-parallel data from monolingual corpora.
After ﬁne-tuning pre-trained self-supervised models with LAgSwAV loss, we sample sentences from both languages and pass them to the model to obtain the SwAV cluster assignment probabilities, which are treated as sentence embeddings.
The soft cluster assignments are then matched by a margin-based algorithm (Artetxe & Schwenk, 2019b) with additional ﬁltering criteria (§3.2).
To augment UMT training with the obtained pseudo-parallel data, we propose a simple ranking-based weighted cross-entropy loss to train on this data along with the standard iterative back-translation (§3.3).
We also adopt a dynamic weight to the augmentation loss to alleviate the overﬁtting effect due to the limited augmentation data.
We tested our method on the standard bilingual WMT’14 English-French, WMT’16 GermanEnglish and English-Romanian unsupervised translation tasks and achieved the state of the art with 40.2, 36.8, and 37.0 BLEU, respectively; which is up to 2 BLEU gain from the baselines.
In the FLoRes low-resource unsupervised setups (Guzm ´ an et al., 2019), our method also outperforms mBART baseline (Liu et al., 2020) by up to 2.8 BLEU in the English-Nepali and English-Sinhala tasks.
We conducted a series of analyses to demonstrate the outperforming ability of LAgSwAV in mining pseudo-parallel data.
The method is also found to outperform supervised LASER (Artetxe &
Schwenk, 2019a), which can be attributed to LASER’s lack of language speciﬁers.
B ACKGROUND 2. U NSUPERVISED M ACHINE T RANSLATION
Lample et al.
(2018a) and Artetxe et al.
(2018) were among the ﬁrst to employ iterative backtranslation in fully unsupervised machine translation (UMT).
Lample et al.
(2018c) later formulated the three-principles to UMT: initialization, language modeling and iterative back-translation (Sennrich et al., 2016a).
Initialization can takes form of MUSE bilingual dictionary (Lample et al., 2018b;a), cross-lingual masked language model (XLM) (Conneau & Lample, 2019) or Seq2Seq model (Song et al., 2019).
Language modeling often takes form of denoising autoencoding (Lample et al., 2018a) or is omitted entirely (Song et al., 2019).
Apart from these principles, Nguyen et al.
(2021) use two distinct UMT teachers to distill the student in a two-stage back-translation process.
Despite being different, most existing methods commonly employ the same iterative backtranslation process.
Multilinguality is also proved to be useful as Liu et al.
(2020) and Garcia et al.
(2021) utilize extra training data of more than two languages.
These methods, however, do not leverage the potential pseudo-parallel data that may exist in the monolingual data to improve UMT
. 2. P SEUDO -P ARALLEL D ATA M INING
Parallel data mining for machine translation has been an active ﬁeld of research (Zweigenbaum et al., 2018; Schwenk, 2018; Chinea-R´ıos et al., 2017).
Artetxe & Schwenk (2019b) suggested an effective margin-based algorithm to mine parallel data using sentence embeddings, which we also use in our method.
More recently, LASER (Artetxe & Schwenk, 2019a) was trained on massive multilingual parallel data from 93 languages.
Nonetheless, most of the effort has been invested largely in a supervised setup where the training data is parallel.
In terms of UMT, Wu et al. (2019a) proposed an extract-edit approach to mine relevant data and edit it for use, while Wu et al.
(2019b) mine weakly-paired documents.
Tran et al.
(2020) suggested to train on massive multilingual data from 25 languages and mine data from 180 translation directions, which may not be applicable in bilingual setup, where monolingual data only for the two languages under consideration are available.
2. S WAPPING A SSIGNMENTS BETWEEN V IEWS (S W AV L OSS )
Caron et al.
(2020) introduced Swapping Assignments between Views (SwAV) loss as a means to conduct self-supervised training in computer vision because masking-based training methods applied in language modeling are intractable in images.
Although SwAV is a type of contrastive learning, it differs from its sisters (Chen et al., 2020;
He et al., 2020; Wu et al., 2018) in that it is an online clustering-based method.
This means the model is trained to assign cluster labels, or “code” vector q , to an input image with encoded latent features z .
In particular, SwAV loss seeks to enforce consistency between codes of different augmentations (or views) of the same image.
In other words, different augmentations of the same image should have almost the same semantic content as the original, and thus the cluster assignments of them should also be consistently the same.
Formally, let z 1 , z 2 ∈ R
d be latent features of two distinct views X 1 , X 2 of the same image X after passing them through an encoder f θ , we compute the corresponding codes q 1 , q 2 ∈ [0 , 1] K by passing z 1 , z 2 into the prototype layer, which consists of prototype weights C
= [ c 1 , ..., c K ] with c i ∈ R d .
We then proceed to swap the codes q 1 , q 2 and use them as labels for z 2 , z 1 , respectively:
[Equation]
The loss function l ( z , q ) in Equation (1) measures the similarity distance between features z and code q , and is deﬁned as:
[Equation]
where τ is a temperature hyper-parameter and k is the index for c k row in C .
Figure 2a illustrates how SwAV loss works.
While the above formulations involve two different augmentations X 1 , Xof the same image for brevity purpose, multiple augmentations are used in practice.
We refer the reader to (Caron et al., 2020) for further details on SwAV loss.
M ETHOD
In this section, we present our three main contributions: the language-agnostic constraint for SwAV loss or LAgSwAV (§3.1), the pseudo-parallel data mining with ﬁlter suite (§3.2), and ﬁnally the rank-based cross-entropy loss for UMT training with augmented data (§3.3).
3. L ANGUAGE -A GNOSTIC C ONSTRAINT FOR S W AV L OSS IN L ANGUAGE M ODEL
The SwAV loss was developed to conduct self-supervised training in vision due to infeasibility of applying masked language modeling (MLM) to images (Caron et al., 2020).
Thus, for language model pre-training, we stick to MLM and use our proposed LAgSwAV to ﬁne-tune a pre-trained
LM to achieve semantic contrastiveness and language agnosticism.
Speciﬁcally, in the presence of sentences from two languages, let B s =
[ X 1 , ..., X B ] and B t =
[ Y 1 , ..., Y B ] be the batches of B sentences from language s and t , respectively.
We proceed to
[Figure]
augment them into B s, 1 =
[ X 11 , ..., X 1 B ] and B s, 2 =
[ X 21 , ..., X 2 B ] , and B t, 1 =
[ Y 11 , ..., Y 1 B ] and
B t, 2 =
[ Y 21 , ..., Y 2 B ] by various noising techniques, such as token swapping, masking and deleting.
Next, we denote F θ as the Transformer encoder (Conneau & Lample, 2019), which outputs the sentence embedding z given an input sentence.
We compute the respective latent features
Z s, 1 , Z s, 2 , Z t, 1 , Z t, 2 ∈ R B × d for B s, 1 , B s, 2 , B t, 1 , B t, 2 using F θ , as described in §2.3.
Subsequently, we compute the code cluster assignments Q s, 1 , Q s, 2 , Q t, 1 , Q t, 2 ∈ [0 , 1] B × K as:
[Equation]
sinkhorn
[Equation]
where [ • ; • ] is the row-wise concatenation function and sinkhorn ( • ) is the iterative Sinkhorn-Knopp algorithm (Cuturi, 2013), a batch-wise renormalization process to compute the cluster assignment probabilities, and (cid:15) is a hyper-parameter to control the smoothness of the distribution.
The above formulation is in fact indifferent from the vanilla SwAV loss (§2.3).
However, intuitively, if the model F θ is not language-agnostic ( i.e., s and t are distributionally separate), the resulting cluster assignments Q s,j and Q t,j are likely to be distributed unevenly for s and t .
In other words, some clusters c i ⊂ C will be frequently assigned to the sentences of language s but rarely to the sentences of t , and vice versa.
Figure 2b illustrates the imbalance phenomenon.
We argue that it is possible to enforce the language-agnostic property indirectly by ﬁxing this imbalance phenomenon.
Therefore, the objective of our proposed LAgSwAV is to measure the stochastic imbalance in cluster assignments Q s,j , Q t,j , and rebalance them into V s,j , V t,j ∈ [0 , 1] B × K such that clusters in
V s,j , V t,j are stochastically assigned evenly regardless of the language of the sentences.
Then, we compute the total LAgSwAV loss as usual for both s and t as:
[Equation]
We formalize the transformation from Q to V using the language-agnostic rebalancing function Ω , which operates at the batch level to capture the statistics on cluster assignments.
Speciﬁcally, given the Q matrices speciﬁed above, we compute the per-language rebalance weight vectors w s , w t ∈ R K , such that when multiplying them with prototype outputs, they try to suppress the magnitudes of frequently-assigned clusters and promote those of rarely-assigned clusters, resulting in more balanced cluster assignments.
More formally, the weight vectors are computed as:
[Equation]
where onehot ( • ) is a row-wise one-hot function which returns a one-hot vector by turning the maximum entry in the row to 1 . 0 and the rest to 0 .
We then use the weight vectors w s , w t to compute the rebalanced cluster assignments V s,j , V t,j ∈ R B × K by modifying Equation (3) as:
[Equation]
s
[Equation]
s
[Equation] [Equation]
where a (cid:12) B is an element-wise multiplication operation of vector a applied to each row of B , and the rows of V s,j , V t,j represent the v s,ji , v s,ji vectors in Equation (4).
Note that there is no gradient ﬂow for the computation of w s and w t , as well as the cluster assignments Q and V .
3. P SEUDO -P ARALLEL D ATA M INING WITH F ILTER S UITE
To mine pseudo-parallel data, we sample sentences from two monolingual corpora s and t , and compute their cluster assignments using our LAgSwAV model.
The cluster assignment probabilities are treated as sentence embeddings in the next step to mine the data using the margin-based algorithm proposed by Artetxe & Schwenk (2019b).
The resulting dataset is decreasingly sorted by the alignment scores.
However, we empirically found that the obtained data are considerably noisy.
Therefore, we propose a ﬁlter suite , which is a set of ﬁltering criteria to ﬁlter out low-quality samples from the initial mined dataset, X r s − t =
[( x 1 , y 1 ) , ..., ( x m , y m )] , resulting in the ﬁnal mined data X s − t =
[( x 1 , y 1 ) , ..., ( x n , y n )] .
In particular, we ﬁlter out by:
Minimum and maximum sentence lengths of L min and L max for both source and target.
Source-target length ratio µ with µ ≥ 1 . 0 ; i.e., 1 µ ≤ len ( x i ) len ( y i ) ≤ µ .
Subword overlap ratio γ i between source and target is larger than the corpus mean value, where γ i is deﬁned as
[Equation]
Accept only top ρ % of pairs based on the alignment scores.
This is equivalent to threshold ﬁltering (Artetxe & Schwenk, 2019b), but it does not depend on an absolute threshold value.
UMT ﬁltering with agreement BLEU > β .
Speciﬁcally, after training a UMT model with the mined data from steps 1-4, we use the model itself to translate each source sentence of the mined data and measure the BLEU score between its translation and the target.
We then ﬁlter out any pair with BLEU <
β .
This step drastically reduces the size of the mined dataset
Further details and motivations for these criteria are given in the Appendix.
3. R ANK - BASED C ROSS -E NTROPY FOR UMT I NTEGRATION
It is trivial to compute a supervised loss on the mined pseudo-parallel data and add it to the standard
UMT back-translation loss.
However, we found that ( i ) the mined data still contains, especially at the bottom of the ranked list, certain amount of misalignments, and ( ii ) the data is too small ( < 100K sentence pairs) to not cause overﬁtting.
These issues motivate us to develop a Rank-based
Cross-Entropy loss, or Rank-XE , to be applied to the mined data.
Speciﬁcally, given a mined dataset X s − t =
[( x 1 , y 1 ) , ..., ( x n , y n )] , which is decreasingly sorted by their alignment scores, we assign a weight σ i = 1 . 0 − ( i − 1)
/n for each pair ( x i , y i ) .
σ i is designed to assign higher weights to the pairs at the top and lower to the bottom ones, which effectively reduces the inﬂuence of low-ranked pairs to the loss function.
The loss for the mined (or augmented) data is then deﬁned as:
[Equation]
which is then added to the standard iterative back-translation loss L BT to compute the total loss:
[Equation]
where the weight λ aug is set to 1 at the beginning and gradually decreases to 0 to avoid overﬁtting.
E XPERIMENTS
In this section, we ﬁrst report the performance of our method in comparison with the established baselines on the standard WMT and FLoRes low-resource UMT tasks.
Then, we experimentally
[Table]
show how LAgSwAV is able to mine better pseudo-parallel data.
We also compare our method with related approaches and conduct other analyses.
To optimize for the most informative content within the limited page real estate, we leave some of experimental details to the Appendix.
4. WMT
T
Setup.
For the WMT’14 English-French (En-Fr), WMT’16 English-German (En-De) and WMT’
English-Romanian (En-Ro) bilingual UMT tasks, we follow the established predecessors (Lample et al., 2018c; Conneau & Lample, 2019;
Song et al., 2019; Nguyen et al., 2021) to use only the monolingual data from 2007-2017 WMT News Crawl datasets of the two languages for each task.
Note that no data from a third language is used.
We ﬁne-tune the pre-trained XLM (Conneau &
Lample, 2019) with our proposed LAgSwAV loss and follow the process in §3.2 to mine pseudoparallel data, which are then used to augment MASS UMT models (Song et al., 2019).
Following all established previous work, we evaluate the models with tokenized BLEU multi-bleu.perl script (Koehn et al., 2007).
In addition, we also report the corresponding sacrebleu scores (Post, 2018) in the subscript for our method as well as the baselines that we reproduced in our experiments.
Results.
From the results reported in Table 1, we notice that our method outperforms MASS baseline (Song et al., 2019) by 2.7, 2.1, 1.2, 1.6, 1.8 and 1.8 BLEU for the WMT En-Fr, Fr-En,
En-De, De-En, En-Ro and Ro-En, respectively.
It also surpasses CBD (Nguyen et al., 2021) by up to 2 BLEU in 5 out of 6 language pairs.
The results demonstrate that the pseudo-parallel data mined by our method is valuable for the UMT models.
4. L OW -R ESOURCE U NSUPERVISED T RANSLATION
[Table]
For the FLoRes low-resource UMT tasks (Guzm´an et al., 2019), we follow the same setup as mBART (Liu et al., 2020).
Speciﬁcally, we reuse their published pre-trained model to ﬁne-tune with our LAgSwAV loss and mine pseudo-parallel data.
Table 2 reports the results on Ne-En and Si-En for our method, as compared with XLM (Conneau & Lample, 2019), MASS (Song et al., 2019) and
mBART (Liu et al., 2020).
It can be shown that our method outperforms mBART by up to 2.8 BLEU in these UMT tasks.
4. U NDERSTANDING L ANGUAGE A GNOSTIC C ONSTRAINT FOR S W AV LOSS
To understand how LAgSwAV behaves in terms of the aforementioned language agnostic and semantically contrastive properties, we conduct a series of evaluations with manual visualization as well as automatic metrics to measure the focused properties.
[Figure]
4.3. T-SNE V ISUALIZATIONS
We sample 50K sentences of both languages from a held-out monolingual data for each language pair and use T-SNE (Van der Maaten & Hinton, 2008) to visualize the sentence embeddings produced by XLM, XLM ﬁne-tuned with SwAV (XLM+SwAV) and XLM ﬁne-tuned with LAgSwAV (XLM+LAgSwAV).
From the visualizations shown in Figure 3, it can be seen that the English embeddings (red) are scattered separately and distinguishable from those of French and German (cyan) in XLM and SwAV XLM.
Among them, the visualizations for German fare better than those of
French.
However, after ﬁne-tuning with LAgSwAV loss, the sentence embeddings are mixed indistinguishably, indicating a high degree of language-agnosticism.
4.3. Q UANTITATIVE M EASURES
Although §4.3.1 provides a qualitative sense of how the models exhibit in terms of the two examined properties, it is necessary to measure the properties in a quantiﬁable manner.
We propose three metrics to evaluate the models: Tatoeba Accuracy, Global Accuracy and Cluster Word Distribution.
We compare different models on these metrics in Table 3.
In the Appendix, we also provide sentence samples of different clusters to show that they are semantically related.
Tatoeba Accuracy.
Artetxe & Schwenk (2019a) proposed to use parts of the Tatoeba corpus as test sets for the Tatoeba similarity search task, which can be used to measure how good a model is at mining parallel data.
Speciﬁcally, for each source sentence, we use cosine similarity to search for the closest target sentence based on the sentence embeddings to check if the obtained target sentence is indeed the correct target of the given source, from which we compute the Tatoeba accuracy score.
Global Accuracy.
This metric uses a validation set of parallel sentences to measure how languageagnostic the models are, based on the perception that non-agnostic models exhibit a separation of latent spaces for different languages.
Speciﬁcally, if there is a separation by language, each sentence of language s would be closer to all other sentence of the same language s than any sentence of language t .
Therefore, we measure Global Accuracy in the same way as Tatoeba except that we search for the target sentence in a search space that includes not just the target sentences, but both source and target sentences combined.
If the model is non-agnostic, the search will always pick incorrect intra-lingual sentences, driving down the accuracy signiﬁcantly.
Word Distribution Score.
This metric uses a held-out monolingual data to measure how semantically contrastive the models are in terms of content and topics.
One of the ways to measure this is to detect certain words that appear frequently in only one cluster and rarely in others.
Intuitively, if a cluster contains certain key-words dominantly, it is likely the sentences in that cluster are of similar semantic topic or content.
From this notion, the word distribution score is computed based on the unique word appearance frequency for each cluster.
Further details are given in the Appendix.
Results.
Table 3 shows the results in terms of the three metrics for the XLM (baseline),
XLM+SwAV, and XLM+LAgSwAV models along with their respective BLEU scores achieved using the mined data obtained by the models after going through the Filter Suite (§3.2).
We observe consistent improvements across all three measures for our LAgSwAV model.
In particular, the global accuracy scores reﬂect closely the degree of language-agnosticism as observed in Figure 3.
For example, the global accuracy values for French and Romanian are particularly low for the base-
[Table] [Table]
lines, while that of German is already much higher.
These improvements also translate well to the performance of our LAgSwAV models on the downstream UMT tasks
[Table]
4. C OMPARISON AND A BLATION S TUDY
As an unsupervised bitext mining method
[Table]
it is noteworthy to compare our method with existing methods (Wu et al., 2019a;b).
We also report the results for LASER (Artetxe & Schwenk, 2019a)
[Table]
Note that LASER is not directly comparable to our setup as it was trained on parallel data fromlanguages.
Speciﬁcally, we use the pre-trained LASER model to mine pseudo-parallel data from the same monolingual sources used in LAgSwAV.
We then use the mined data to augment UMT models with and without the Filter Suite (§3.2) and rank-based loss (§3.3).
Plus, we also conducted an ablation study to examine the contributions of different components of our method.
We train the
UMT models with the mined data from our LAgSwAV models without any ﬁltering and rank-based loss, and then subsequently add these components to see how they contribute to the performance.
Ablation Study.
From the results in Table 4, it is noticeable that without a sufﬁcient ﬁltering process, the mined data by LASER as well as LAgSwAV cause signiﬁcant performance drops across all language pairs.
This is because the majority of sentence pairs ranked at the bottom of the mined data are noisy, causing incorrect loss terms.
The performances gradually improve as we add rankbased cross entropy and dynamic λ aug weight to the framework, which essentially reduce the impact of low-ranked pairs.
However, the most performance jumps are observed when we add the Filter
Suite, where we considerably ﬁlter the mined data and retain the top ≈ 5% of the mined data.
These observations are true for both LASER and our LAgSwAV ﬁne-tuned XLM models.
Comparison with LASER.
A more surprising observation we have from the results in Tableis that LASER underperforms our method, even with the Filter Suite.
This is despite the fact that
LASER was trained on a much larger parallel data from 93 languages.
After investigation, we discover that LASER does not differentiate languages and the monolingual data contains sentences from the other language.
For instance, the English corpus contains some French sentences as artifacts; LASER simply picks these artifacts up and pair them with their identical matches found in the French corpus.
Since these Fr-Fr pairs are almost identical, they are (wrongly) ranked on top of
[Table]
the mined dataset.
Our LAgSwAV, meanwhile, differentiates languages thanks to XLM’s language embedding layer.
Therefore, when it encounters a French sentence that is ‘marked’ as English, the model will infer it as erratic and avoid selecting it during mining.
Table 5 veriﬁes this observation.
4. F URTHER A NALYSIS
Quality of Mined Data.
To gain further insights into the quality of the augmentation dataset mined by our method, we use a supervised NMT model (Ott et al., 2018) to translate the source side of the mined data and compute the agreement BLEU scores between its translations and the dataset’s target side.
The higher the agreement BLEU, the more likely the source and target of the mined dataset are semantically and meaningfully aligned, according to the NMT model.
We use this metric to compare the quality of mined datasets produced by XLM, supervised LASER (Artetxe & Schwenk, 2019a),
XLM+SwAV and XLM+LAgSwAV.
As shown in Table 5, XLM and XLM+SwAV fail completely at mining pseudo-parallel data as their agreement BLEU scores are signiﬁcantly low.
Meanwhile, our method outperforms even supervised LASER, which aligns well with the results in Table 4.
Percentage ρ % .
Among the hyper-parameters speciﬁed in the Filter Suite (§3.2), we found that the ρ % plays the paramount role.
As shown in Figure 4, for the WMT En-Ro task, we found only the top 10% of the mined data is valuable to improve UMT performance.
As we gradually include more data from the bottom of the mined data by increasing ρ , the performance degrades substantially.
[Figure] [Figure]
[Table]
Translationese Effect.
Edunov et al.
(2020) found that back-translation only beneﬁts translationesesource sentences but does not improve for natural-source texts.
To investigate whether our method is affected by the translationese effect , we use the test sets provided by Edunov et al.
and replicate the translationese analysis in three setups: ( i ) Natural-source → Translationese-target (X → Y*), ( ii ) Translationese-source → Natural-target (X* → Y) and ( iii ) Translationese of translationese of source → Translationese-target (X** → Y*).
The results in Table 6 show that our method improves the performance in De-En and En-De UMT tasks for both natural-source and translationese-source setups.
This demonstrates that our method is not affected by the translationese
C ONCLUSION
We have proposed a new language-agnostic SwAV loss, which is used to ﬁne-tune pre-trained language models to improve their ability to mine pseudo-parallel data in an unsupervised manner.
We have also introduced additional modules, such as Filter Suite and dynamic loss weight, to improve the performance of UMT systems.
The method achieves signiﬁcant improvements over the baselines in the WMT English-French, German-English and English-Romanian bilingual UMT tasks.
It also outperforms mBART in the FLoRes Nepali-English and Sinhala-English UMT tasks.
Our analysis also shows that the method outperforms the baselines in mining pseudo-parallel data.
E THICS AND R EPRODUCIBILITY S TATEMENTS
Ethics.
The paper presents research on unsupervised machine translation where experiments were conducted on publicly available datasets, such as the NewsCrawl and CommonCrawl datasets, which have been used in various established publications.
Thus, we believe the research and experiments do not contain any questionable concern regarding bias, discrimination, fairness, privacy, etc.
Nonetheless, in case of application to production systems, we advise practitioners to use our solutions with caution, as our mining method may still align source-target pairs that are close but not exact, such as “I am from the US” versus “I am from California”.
These inexact pairs may then inﬂuence the downstream translation model to make incorrect predictions.
Reproducibility.
Apart from the experimental explanations in the main paper, Appendix appendix A.3 provides further experimental details on the WMT and FLoRes unsupervised translation tasks, as well as other analysis experiments.
In addition, the source codes provided above contain further technical instructions and code explanations to reproduce the results.
A A PPENDIX A. F URTHER E XPLANATION FOR O UR M ETHOD A.1. L ANGUAGE -A GNOSTIC C ONSTRAINT FOR S W AV L OSS IN L ANGUAGE M ODEL
In addition to the details explained in the main paper we illustrate the batch-wise sinkhorn algorithm (Cuturi, 2013) in Algorithm 1.
Further details about this procedure and the SwAV loss are also provided in Caron et al.
(2020) output Q ∈ R B × K .
1: Q ← Z T 2: Q ← (cid:80) i (cid:80) j Q i,j 3: U , R , C ← zeros ( K ) , ones ( K ) /K, ones ( B ) /B 4: for i in 0...
n do 5:
U ← (cid:80) j Q : ,j 6:
Q ← Q ∗ ( R / U ) 7:
Q ← Q ∗ ( C / ( (cid:80) i Q i, : ))
8: end for 9: Q ← ( Q / ( (cid:80) i Q i, : ))
T 10: return Q
A.1. P SEUDO -P ARALLEL D ATA M INING WITH F ILTER S UITE
In addition to explanation given in the main paper we further clarify certain note-worthy details of the Filter Suite.
Firstly we sample sentences from two monolingual corpora s and t and compute their cluster assignments using our LAgSwAV model with softmax operation instead of sinkhorn because it is unreliable to conduct batch-wise operation during inference.
Speciﬁcally we compute the mining sentence embeddings e X of a given sentence X of any language as:
[Equation]
11) e X is then used by the margin-based algorithm proposed by Artetxe & Schwenk (2019b) to mine pseudo-parallel data.
Secondly, as the proposed Filter Suite consists of a set of different ﬁltering criteria,, we specify the motivations for each of them as follow:
Minimum and maximum sentence lengths of L min and L max : short sentences may not be useful, while long sentences may be more likely to mis-aligned.
Source-target length ratio µ : source and target sentences should be of comparable length, the variation may depend on the speciﬁc language pair.
Subword overlap ratio γ i between source and target: for relatively close language pairs (En, Fr, De, Ro), parallel sentence pairs of the same meanings are likely to share a certain amount of subword; so we accept aligned pairs whose γ i is higher than the corpus average.
In addition, we also cap this value up to a maximum, such as 0.35, to avoid the exact match scenario, where source and target are exactly the same.
Note that we do not use this criterion for low-resource languages, such as Nepali and Sinhala. 4.
Accept only top ρ % of pairs: sentence pairs are ranked based on alignment scores, as a result, pairs with low scores are ranked at the bottom and are often low-quality.
Thus, we ﬁlter them out.
UMT ﬁltering: once the UMT model has been trained to be competent enough, it can act as a ﬁlterer too.
Therefore, we use it to translate the mined data to verify whether the source and target are sufﬁciently aligned.
A.1. R ANK - BASED C ROSS -E NTROPY FOR UMT I NTEGRATION
In addition to the total loss L formulation for the rank-based cross-entropy loss and the UMT integration strategy in the main paper, we further provide the overall UMT integration of the mined dataset procedures in Algorithm 2. Algorithm
1: θ ← ˆ θ 2: while until convergence do 3:
Sample batches X s from X s 4:
Sample batches ( X, Y ) from X s − t , where X =
[ x 1 , ..., x n ] , Y =
[ y 1 , ..., y n ] 5:
L BT = − log P θ ( X s | translate θ ( X s ))
{ Standard BT loss } 6:
L AUG =
(cid:80) i − σ i log P
θ ( y i | x i ) { Augmentation loss } 7:
L = L BT + λ aug L AUG 8: θ ← θ − η ∇ θ L θ 9: end while 10: return θ
A. C OMPLETE LA G S W AV M ODEL
Figure 5 illustrates the complete structure of our LAgSwAV model.
Speciﬁcally, on the left side, we have our input data, which are English ( X ) and Romanian ( Y ) sentences.
These inputs are perturbed into four different augmentations ( X 1 , X 2 , Y 1 and Y 2 ).
These augmentations are then passed to the
Transformer encoder and then the prototype layer, as well as the rebalancing block Ω to compute the ﬁnal LAgSwAV loss.
A. E XPERIMENTAL D ETAILS
WMT bilingual unsupervised tasks.
For the WMT’14 English-French (En-Fr), WMT’16 EnglishGerman (En-De) and WMT’16 English-Romanian (En-Ro) bilingual unsupervised translation tasks, we follow established predecessors (Lample et al., 2018c; Conneau & Lample, 2019; Song et al., 2019; Nguyen et al., 2021) to use the all the monolingual data from 2007-2017 WMT News Crawl datasets of the two languages for each task, which include 190M, 78M, 309M and 3M sentences for English (En), French (Fr), German (De) and Romanian (Ro) respectively.
We ﬁne-tune the pretrained XLM (Conneau & Lample, 2019) provided by the authors with our proposed LAgSwAV loss.
For each language pair, we also use the respective provided bilingual dictionary of 60K subword units (Sennrich et al., 2016b).
Regarding the mining process, we sample up to 20M sentences from the monolingual data explained above for each language and use our LAgSwAV model to mine pseudo-parallel data with Filter Suite explained in §3.2.
We set mininum, maximum lengths of L min = 5 and L max = 300 ; source/target length ratio µ 1.5; maximum overlap ratio γ i = 0 .and
accept only the top ρ = 5% of mined pairs.
The agreement BLEU threshold is β =
We use the mined data to augment the MASS (Song et al., 2019) UMT models.
Following all established previous work, we evaluate the models using tokenized BLEU with Moses multi-bleu.perl script (Koehn et al., 2007).
In addition, we also report corresponding sacrebleu scores (Post, 2018) in the subscript of Table 1 for our method as well as baselines that we can reproduce.
In addition, we also use bootstrap sampling (Koehn, 2004) to test for statistical signiﬁcance for the WMT bilingual unsupervised tasks.
As a result, we round that our method’s improvements are signiﬁcant with p-value < 0.05 for En-Fr, En-De, and En-Ro (both directions).
Plus, we also ran our experiments 3 times and found the standard deviations are approximately 0.1, 0.2, 0.1, 0.3, 0.2, and 0.3 BLEU for En-Fr, Fr-En, En-De, De-En, En-Ro and Ro-En, respectively.
FLoRes low-resource tasks.
For the FLoRes low-resource unsupervised translation tasks (Guzm´an et al., 2019), we follow the same setup as mBART (Liu et al., 2020).
Speciﬁcally, we reuse their published pre-trained mBART model to ﬁne-tune LAgSwAV loss as well as initial model for UMT back-translation training.
We also use the same sentencepiece tokenizer model (Kudo & Richardson, 2018) as mBART.
We sample up to 20M sentences of the English, Nepali and Sinhala corpora of the
CC25 pre-training data to mine pseudo-parallel data and train the UMT model.
To avoid high-rank mis-alignments in the mined data, we ﬁlter the mining monolingual data to include only sentences with unicode \ p { InDevanagari } and \ p { InSinhala } word counts more than 80% the sentence length for Nepali and Sinhala respectively.
T-SNE Visualization.
We sample 50K sentences of both languages from a held-out monolingual data for each language pair and use T-SNE (Van der Maaten & Hinton, 2008) to visualize the sentence embeddings of those sentences produced by the baseline XLM (encoder output of ‘[eos]’ token) and the SwAV as well as LAgSwAV ﬁne-tuned XLM models (softmax representations of prototype outputs softmax( C T Z ) ).
These visualizations are shown in Figure 6.
As it can be seen, the sentence embeddings of English sentences (red) are scattered separately and distinguishable from those of French, German and Romanian (cyan) in XLM and SwAV XLM.
Among these, the visualizations for German fare better than French’s and Romanian’s.
However, after ﬁne-tuning with
LAgSwAV loss, the sentence embeddings are mixed indistinguishably, indicating a high degree of language-agnostic property.
[Figure]
Tatoeba Accuracy
Artetxe & Schwenk (2019a) proposed to use parts of the Tatoeba corpus as test sets for the Tatoeba similarity search task, which can be used to measure how good a model is at mining parallel data.
Speciﬁcally, for each source sentence, we use cosine similarity to search for the closest target sentence based on the sentence embeddings to check if the obtained target sentence is indeed the correct target of the given source, from which we compute the Tatoeba accuracy score.
In other words, given a parallel test set [( X 1 , Y 1 ) , ..., ( X n , ..., Y n )] and a nearest neighbor search function N operating on the sentence embeddings of the sentences, we compute the Tatoeba accuracy as:
[Equation]
Global Accuracy.
This metric also uses a valid parallel set to measure how lang-agnostic the models are, based on the perception that non-agnostic models exhibit a separation of latent spaces for different languages, like in Figure 3.
Speciﬁcally, if there is a separation by language, each sentence of language s would be closer to all other sentence of the same language s than any sentence of the other language t .
Therefore, we measure Global Accuracy in the same way as Alignment Accuracy, except that we conduct search for the target sentence among a search space of not just the target side, but both source and target side combined.
If the model is non-agnostic, the search will always pick incorrect intra-lingual sentences, driving down the accuracy signiﬁcantly.
Formally, with the same notations as above, Global Accuracy is computed as:
[Equation]
Word Distribution Score.
This metric uses a held-out monolingual data to measure how semantically classiﬁable the models are in terms of content and topics.
One of the ways to measure this is to detect certain words that appear frequently in only one cluster and rarely in others.
The intuition is that if a cluster contains certain key-words dominantly, it is likely the sentences in such cluster are of similar semantic topic or content.
Speciﬁcally, given same data sampled in §4.3.1, we use the K-means clustering (Vassilvitskii & Arthur, 2006) to cluster uhthe sentences into N = 20 clusters.
Then, for each unique word w i ∈ V in the data, we count its appearance frequency F jw i for each cluster j .
After divide it by the total occurance of the word w i in all clusters and compute the normalized mean scores for each word.
Formally, the Word Distribution Score is computed as:
[Equation]
Percentage ρ % and Translationese effect.
In addition to analyses of percentage ρ % for En-Ro and translationese effect for En-De, we also report the similar analyses for Ro-En and De-En, respectively.
The results are shown in Figure 7, we observe that the performance of our method for unsupervised Ro-En task reduces gradually with the percentage ρ % .
As shown similarly in Table 7, our method is also not affected by the translationese effect for De-En direction.
[Figure]
[Table]
A.3. C LUSTER S AMPLES
Semantic contrastiveness reﬂects the model’s ability to classify sentences of similar topics and contents into the same cluster group.
To demonstrate this capability of our LAgSwAV model, we randomly sample monolingual sentences from the NewsCrawl English-Romanian corpora and compute their cluster assignments using our LAgSwAV model.
Then, we use a simple K-means clustering method (Vassilvitskii & Arthur, 2006) to cluster the sentences into 20 clusters.
After that, we extract randomly some sentences from these clusters and report them in Table 8.
As it can be seen in the next page , the sentences in each cluster are closely related semantically.
Speciﬁcally, sentences of cluster 1 talk about football , while those of cluster 5 talk about airlines and aviation .
[Table]
appearstobedesignedtolaunch
abigger
[Table]