Improving Parallel Data Identification using Iteratively Refined Sentence Alignments and Bilingual Mappings of Pre-trained Language Models (2020)

Abstract: The National Research Council of Canada’s team submissions to the parallel corpus filtering task at the Fifth Conference on Machine Translation are based on two key components: (1) iteratively refined statistical sentence alignments for extracting sentence pairs from document pairs and (2) a crosslingual semantic textual similarity metric based on a pretrained multilingual language model, XLM-RoBERTa, with bilingual mappings learnt from a minimal amount of clean parallel data for scoring the parallelism of the extracted sentence pairs. The translation quality of the neural machine translation systems trained and fine-tuned on the parallel data extracted by our submissions improved significantly when compared to the organizers’ LASER-based baseline, a sentence-embedding method that worked well last year. For re-aligning the sentences in the document pairs (component 1), our statistical approach has outperformed the current state-of-the-art neural approach in this low-resource context.

## introduction the aim of the fifth conference on machine translation (wmt20) shared task on parallel corpus filtering @xcite ) is essentially the same as the two previous editions @xcite : identifying high-quality sentence pairs in a noisy corpus crawled from the web using paracrawl @xcite , in order to train machine translation (mt) systems on the clean data. this year, the low-resource language pairs being tested are khmer-english (km-en) and pashto-english (ps-en). specifically, participating systems must produce a score for each sentence pair in the test corpora indicating the quality of that pair. then samples containing the top-scoring 5m words are used to train mt systems. while using the filtered parallel data to train a fairseq @xcite neural machine translation (nmt) system remains the same as last year, the organisers are no longer building statistical machine translation (smt) systems as part of the task evaluation. instead, as an alternative evaluation, the filtered parallel corpus is used to fine-tune an mbart @xcite pretrained nmt system. participants were ranked based on the performance of these mt systems on a test set of wikipedia translations @xcite , as measured by bleu @xcite . a few small sources of parallel data, covering different domains, were provided for each of the two low-resource languages. much larger monolingual corpora were also provided for each language (en, km and ps). in addition to the task of computing quality scores for the purpose of filtering, there is also a sub-task of re-aligning the sentence pairs from the original crawled document pairs. cleanliness or quality of parallel corpora for mt systems is affected by a wide range of factors, e.g., the parallelism of the sentence pairs, the fluency of the sentences in the output language, etc. previous work @xcite showed that different types of errors in the parallel training data degrade mt quality in different ways. crosslingual semantic textual similarity is one of the most important properties of high-quality sentence pairs. @xcite scored cross-lingual semantic textual similarity in two ways, either using a semantic mt quality estimation metric, or by first translating one of the sentences using mt, and then comparing the result to the other sentence, using a semantic mt evaluation metric. at the wmt18 parallel corpus filtering task, @xcite 's supervised submissions were developed for the same mt evaluation pipeline using a new semantic mt metric, yisi-1 @xcite (see also section 2.3). at the wmt19 parallel corpus filtering task, bernier-colborne and @xcite exploited the quality estimation metric yisi-2 using bilingual word embeddings learnt in a supervised manner @xcite from clean parallel training data or a weakly supervised manner @xcite from bilingual dictionary. @xcite further showed that using yisi-2 with multilingual bert @xcite on fully unsupervised parallel corpus filtering (i.e. without access of any parallel training data) achieved similar results to those in bernier-colborne and @xcite . this year, the national research council of canada (nrc) team submitted one system to the parallel corpus filtering task and one to the alignment task. the two systems share the same components in scoring the parallelism of the noisy sentence pairs, i.e., the pre-filtering rules and the quality estimation metric yisi-2. for the parallel corpus aligning task, we use an iterative statistical alignment method to align sentences from the given document pairs before passing the aligned sentences to the scoring pipeline. our internal results show that mt systems trained on pre-aligned sentences filtered by our scoring pipeline outperform those trained on the organizers' laser-based baseline @xcite ) by 0.2-1.4 bleu. training mt systems on re-aligned sentences using our iterative statistical alignment method achieve further gains of 0.3-1.8 bleu. ## system architecture there are a wide range of factors that determine whether a sentence pair is good for training mt systems. some of the more important properties of a good training corpus include: in previous years, we explicitly tried to maximize all four of these properties, but this year we focused only on the first two in the scoring presented in section 2.3 below. ## experiments and results we used the software provided by the task organizers to extract the 5m-word samples from the original test corpora according to the scores generated by each alignment and/or filtering system. we then trained a fairseq mt system or fine-tuned an mbart pretrained nmt using the extracted subsamples. the mt systems were then evaluated on the official dev set ("dev-test"). we exhaustively experimented with the last few layers of both xlm-roberta base and xlm-roberta large in order to find out the model and layer best representing crosslingual semantic similarity. figure 1 shows the plots of the change in bleu scores of each mt system using the embeddings extracted from the n th layer, counting from the last, of the multilingual lm for evaluating crosslingual lexical semantic similarity. in general, we see a trend of rising performance as we roll back from the last layer. the performance peaks at some point and starts to fall when we roll back too far from the end. for xlm-roberta base , the peak performance of the mt systems is achieved by the 3 rd or 4 th last layer (out of 12 layers). for xlm-roberta large , the peak performance of the mt systems is achieved by the 8 th last layer (out of 24 layers). the peak performance of mt systems trained on sentences filtered by xlm-roberta large based yisi-2 is better than that by xlm-roberta base based yisi-2. next, we see that using yisi-2 with xlm-roberta large 's 8th last layer as parallelism scoring function outperforms the laser baseline by 0.1-0.9 bleu in different translation directions and mt architectures. this is our "nrc.baseline" system, and the baseline used for filtering the noisy corpus in training the ibm-hmm alignment model for the "nrc.alignment" system. adding the lm score to the scoring function shows small improvements. learning the cross-lingual linear projection matrix to transform the source embeddings in the target language subspace shows more improvements overall. this is our "nrc.filtering" submission to the parallel corpus filtering task. at last, we show that using our iterative statistical alignment method to redo the alignment of sentences from the given document pairs improves the translation quality of the resulting mt systems significantly. this is our "nrc.alignment" submission to the parallel corpus filtering task. ## conclusion and future work in this paper, we presented the nrc's two submissions to the wmt20 parallel corpus filtering and alignment for low-resource conditions task. our experiments show that yisi-2 is a scoring function of parallelism that is very competitive, and that a statistical sentence alignment method is still able to provide better alignment results than neural ones in low resource situations. further analysis is required to understand the characteristics of the sentence pairs aligned by the baseline vecalign and our iterative statistical sentence alignment and how the latter achieves better translation quality for the trained mt systems. it is worth highlighting that in this task, as well as in our inuktitut-english corpus alignment work @xcite , a well-tuned statistical sentence-alignment system outperformed a stateof-the-art neural one. we hypothesise that this is a low-resource effect, but further work is still needed to explore the best low-resource corpus alignment methods. in particular, we intend to integrate yisi-2 into our sentence aligner to test whether it's our iterative alignment methodology that makes the difference or the fact that the underlying scoring function is statistical (we use ibm-hmm models for sentence pair scoring in our aligner). it's possible that the statistical approach might continue to win here, because in the low-resource context there might not be enough training data to tune the orders of magnitude more parameters of the neural models; a counter-argument is that yisi-2 did better on the scoring task than statistical scoring functions. our future work will explore the tradeoffs between these two approaches, and consider hybrid methods.