Bicleaner at WMT 2020: U niversitat d’Alacant-Prompsit’s submission to the parallel corpus filtering shared task (2020)

Abstract: This paper describes the joint submission of Universitat d’Alacant and Prompsit Language Engineering to the WMT 2020 shared task on parallel corpus filtering. Our submission, based on the free/open-source tool Bicleaner, enhances it with Extremely Randomised Trees and lexical similarity features that account for the frequency of the words in the parallel sentences to determine if two sentences are parallel. To train this classifier we used the clean corpora provided for the task and synthetic noisy parallel sentences. In addition we re-score the output of Bicleaner using character-level language models and n-gram saturation.

## introduction this paper describes the joint submission of universitat d'alacant and prompsit language engineering to the parallel corpus filtering shared task at the fifth conference on machine translation (wmt 2020). our submission is built upon bicleaner @xcite ), foot_0 a widely-used free/open-source tool for detecting noisy parallel sentences that participated in the 2018 edition of this shared task and ranked fourth out of 17 submissions on one of the sub-tasks. we provide quality scores for the sentence pairs provided by the organiser without re-aligning them. the 2020 edition of the parallel corpus filtering shared task focuses on two under-resourced asian languages paired with english: khmer and pashto. khmer (km) is the official language of cambodia and is spoken circa 16 million people in cambodia, vietnam and thailand. 2 there are about 500k english-khmer parallel sentences in opus, foot_2 mainly belonging to narrow domains like software products and religion. pashto (ps) is spoken by around 40 million people in pakistan and in afghanistan, where it is official together with persian. 4 there are around 100k english-pastho parallel sentence in opus, most of which belong to the software domain. detecting noisy parallel sentences for underresourced language pairs, like those addressed in this shared task, is challenging. pastho is not directly supported by laser @xcite , although it supports other iranian languages, and there are few bilingual resources for building the bicleaner's models. bicleaner is based on a classifier that assesses whether a pair of sentences are mutual translations or not. it is trained on a parallel corpus (positive samples) and on an automatically corrupted version of the same corpus (negative samples). the most important features used by the classifier are lexical similarity scores obtained with the help of probabilistic bilingual dictionaries, which are also extracted from the parallel corpus. our submission improves the performance of the version of bicleaner that took part in the 2018 shared task in multiple ways: a new classification algorithm, new lexical features that account for the frequency of the words in the parallel sentences, and a novel way of generating corrupted pairs of sentences. in addition, we re-score the output of bicleaner combining character-level language models and an n-gram saturation scorer in a linear combination whose parameters are determined by fine-tuning the mbart model provided by the organisers of the shared task. the rest of the paper is organised as follows. section 2 describes the bicleaner classifier whereas section 3 explains how the score produced by the classifier is combined with the information provided by character-level language models and an n-gram saturation algorithm to produce the submitted score. section 4 then describes the process followed to build the submission, and section 5 lists related approaches. the paper ends with some concluding remarks. ## bicleaner classifier bicleaner is based on an automatic classifier that produces a score for a pair of sentences representing the probability that they are mutual translations. random forests @xcite , the classification algorithm used in the 2018 submission, has been replaced by extremely-randomised trees @xcite because the latter performed best on preliminary experiments. the extremely randomised trees classification algorithm works by selecting at each internal node the best feature from a sub-set of features selected at random from the whole set of features f , and using a random cut-off point. the hyper-parameters controlling the training of these classifiers are therefore the method used to rank the features and select the best one, the size of the subset of features selected at random, and the number of trees to be used. to select the best hyper-parameters we performed a grid search with the following hyperparameter values. for the ranking we tried with gini importance @xcite ) and information gain; for the size of the sub-set of features we tried with |f |, log 2 |f | and |f |; for the number of trees we tried with 100, 200, 300, 400 and 500. the features we used can be split in two groups: those that account for the lexical similarity of the two sentences, and those based on shallow properties of the sentences. ## re-scoring subsampling 5 million words from the raw corpus based on the score described in the previous section ensures that nmt systems are trained on parallel data. however, some of the selected training parallel samples may not bring useful information and replacing them with other, more informative samples could improve the performance of the resulting nmt systems. we hypothesise that two main reasons could make a pair of sentences which are mutual translations non-informative: i) sentences are not fluent enough and hence very different from those that will be translated with the resulting nmt systems: lists of keywords or website menus are examples of such non-fluent sentences; and ii) the pair of sentences is too similar to other training samples. to take into account these additional factors, the final score assigned to each sentence pair was computed as follows. first, each sentence received a preliminary score, prescore, computed as: after computing prescore, sentence pairs were sorted by that score in descending order, and the score of those pairs for which all their 3-grams could be found in sentences with a higher score was multiplied by a penalty β to promote diversity in the subsampled corpus. the values of the parameters λ and β, that control the contribution of parallelness, fluency and novelty to the final score were optimised so as to maximise the bleu score obtained after fine tuning the mbart model provided by the task organisers. the nelder-mead algorithm @xcite , which does not require gradient computations, was used. ## building the submission this section describes the process followed to build our submission, which comprised selection of training data, corpora preprocessing, classifier training and evaluation of different alternatives for some of the steps. ## related work a shared task on parallel corpus filtering was part of the wmt conference programme for the first time in 2018 @xcite . tl sentence given an sl sentence, emerged as the dominant approach (junczys-dowmunt, 2018). last year's edition was focused on a lowresource scenario @xcite , where parallel data big enough to build nmt models that provide reliable tl probability distributions was not available. the best performing model was laser, a method based on multilingual sentence embeddings @xcite that takes advantage of the data available for multiple language pairs. in fact, a laser model trained on 93 languages is the baseline model published by the organisers for this edition of the shared task. unlike laser, our submission is mainly based on lexical similarity scores analogous to those used in statistical machine translation. they are computed only on parallel data, without any kind of transfer learning from other language pairs. the approach we follow to detect sentences that are mutual translations is similar to the one by @xcite for detecting parallel sentences in comparable corpora. however, we use a larger set of shallow features not related to lexical similarity and follow a more sophisticated method for generating negative samples. concerning our re-scoring strategy for including information about fluency and diversity, participants from past editions also used these attributes to score sentences. for instance, @xcite and @xcite devised a scoring strategy under the assumption that parallel sentences should have similar monolingual language model perplexities, and many other submissions included a penalty for repetitive sentences @xcite @xcite . nevertheless, to the best of our knowledge, our approach is the first one that directly optimises the weight of these attributes towards an automatic translation evaluation metric. ## concluding remarks we described the joint submission of universitat d'alacant and prompsit language engineering to the parallel corpus filtering shared task at the fifth conference on machine translation (wmt 2020). our submission is based on bicleaner, an open source tool based on a classifier that uses lexical similarity features inspired in the translation probabilities used in statistical machine translation. we presented a series of improvements over the version of bicleaner that participated in the 2018 edition of the shared task, namely a better classifier, more sophisticated generation of negative samples and a reformulation of the lexical similarity scores which takes into account word frequency. we showed that these improvements are effective and they allowed our submission to outperform laser, a state-of-the-art method based on multilingual sentence embeddings. moreover, combining bicleaner scores with scores that account for fluency and diversity further improved the results. we plan to keep exploring subword segmentation algorithms that help to fight data sparseness when computing lexical similarity scores with the help of bilingual dictionaries. we also aim at integrating word embeddings into lexical similarity scores, which would allow us to leverage monolingual data in a more effective way.