Can Synthetic Translations Improve Bitext Quality? (2022)

Abstract: Synthetic translations have been used for a wide range of NLP tasks primarily as a means of data augmentation. This work explores, instead, how synthetic translations can be used to revise potentially imperfect reference translations in mined bitext. We find that synthetic samples can improve bitext quality without any additional bilingual supervision when they replace the originals based on a semantic equivalence classifier that helps mitigate NMT noise. The improved quality of the revised bitext is confirmed intrinsically via human evaluation and extrinsically through bilingual induction and MT tasks.

## introduction while human-written data remains the gold standard to train neural machine translation (nmt) and multilingual nlp models, there is growing evidence that synthetic bitext samples-sentencepairs that are translated by nmt-benefit a wide range of tasks. they have been used to enable semi-supervised mt training from monolingual data @xcite @xcite , to induce bilingual lexicons @xcite , and to port models trained on one language to another @xcite . while synthetic bitexts are useful additions to original training data for downstream tasks, it remains unclear how they differ from naturally occurring data. some studies suggest that synthetic samples might be simpler and easier to learn @xcite . recognizing that naturally occurring bitext can be noisy, for instance, when they are mined from comparable monolingual corpora @xcite @xcite , we hypothesize that synthetic bitext might also directly improve the equivalence of the two bitext sides. thus synthetic samples might be useful not only for data augmentation but also to revise potentially noisy original bitext samples. in this paper, we present a controlled empirical study comparing the quality of bitext mined from monolingual resources with a synthetic version generated via mt. we focus on the widely used wikimatrix bitexts for a distant (i.e, @xcite and a similar language-pair (i.e, en-ro), since it has been shown that this corpus contains a significant proportion of erroneous translations @xcite . we generate synthetic bitext by translating the original training samples using mt systems trained on the bitext itself and therefore do not inject any additional supervision in the process. we also consider selectively replacing original samples with forward and backward synthetic translations based on a semantic equivalence classifier, which is also trained without additional supervision. we show that the resulting synthetic bitext improves the quality of the original intrinsically using human assessments of equivalence and extrinsically on bilingual induction (bli) and mt tasks. we present an extensive analysis of synthetic data properties and of the impact of each step in its generation process. this study brings new insights into the use of synthetic samples in nlp. first, intrinsic evaluation shows that synthetic translations, in addition to "normalizing" the bitext @xcite , could potentially provide reference translations that are more semantically equivalent to the source than the original ones. furthermore, the improved bitext provides more useful signals for bli tasks and nmt training in two settings (training from scratch; continued training), as confirmed by our extrinsic evaluations. finally, ablation analyses that compare different ways to combine synthetic translations show that using both translation directions and filtering using semantic equivalence is key to improving bitext quality and calls for further exploration of best practices for using synthetic translations in nlp tasks. ## approach this section describes the methods and data we use to produce revised bitexts for our empirical study. ## extrinsic evaluation of bitext quality our previous analysis suggests that selective replacement of divergent references with synthetic translations results in bitext of improved quality, with reduced level of noises and easier word-level mappings between the two languages, when compared to the original wikimatrix corpus. to better understand how those differences impact downstream tasks, we contrast the improved bitext with the original through a series of extrinsic evaluations for en-el and en-ro languages that rely on parallel texts as training samples (see §5.2). first, we focus on the recent state-of-the-art unsupervised bli approach of shi et al. (2021) that relies on word-alignments of extracted bitexts. second, we follow the recent bitext quality evaluation frameworks adopted by the "shared task on parallel corpus filtering and alignment" @xcite and built neural machine translation systems from scratch and by continued training on a multilingual pre-trained transformer model. finally, we conduct extensive ablation experiments to test the impact of using synthetic translations without the semantic equivalence condition and contrast with familiar techniques used by prior work (see §5.3). ## conclusion this paper explored how synthetic translations can be used to revise bitext, using nmt models trained on the exact same data we seek to revise. our extensive empirical study surprisingly shows that, even without access to further bilingual data or supervision, this approach improves the quality of the original bitext, especially when synthetic translations are generated in both translation directions and selectively replace the original using a semantic equivalence criterion. specifically, our intrinsic evaluation showed that synthetic translations are of sufficient quality to improve over the original references, in addition to "normalizing" the bitext as suggested by prior work and corpus level statistics @xcite . extrinsic evaluations further show that the replaced synthetic translations provide more useful signals for bli tasks and nmt training in two settings (i.e., training from scratch and continued training). these findings provide a foundation for further exploration of the use of synthetic bitext. first, we focused our empirical study on language pairs and datasets where revising bitexts is the most needed and most likely to be useful: the resources available for these languages are not so large that mined bitext can simply be ignored or filtered with simple heuristics, yet there is enough data to build nmt systems of reasonable quality (i.e., ∼ 600k segments for en-ro, and ∼ 750k for en-el). while in principle, selective replacement of divergent references with synthetic translations should port to high-resource settings, where nmt is as good or better than for the languages considered in this work, other techniques are likely needed in low-resource settings where nmt quality is too low to provide reliable candidate translations. second, having established that the revised bitext improves the quality of the original bitext in isolation, it remains to be seen how to best revise bitexts in more heterogeneous scenarios with diverse sources of parallel or monolingual corpora. overall, as synthetic data generated by nmt is increasingly used to improve crosslingual transfer in multilingual nlp, our study motivates taking a closer look at the properties of synthetic samples to better understand how they might impact downstream tasks beyond raw performance metrics. all bitexts are available at: https: