An exploratory approach to the Parallel Corpus Filtering shared task WMT 20 (2020)

Abstract: In this document we describe our submission to the parallel corpus filtering task using multilingual word embedding, language models and an ensemble of pre and post filtering rules. We use the norms of embedding and the perplexities of language models along with pre/post filtering rules to complement the LASER baseline scores and in the end get an improvement on the dev set in both language pairs.

## introduction for this task the participants were provided with a corpus of parallel data in pashto-english (ps-en) and khmer-english (km-en). additional parallel and monolingual datasets were also provided. the task organizers built neural machine translation (nmt) systems from the scores produced, based on parallel training sets of 5 million words. these systems are sensitive towards noise @xcite and thus, it becomes important to separate the useful data from the noise. we view the task as data that passes through a pipeline of filters in order to give us the best possible selection of 5 million words in the end. we determined that language id filtering is a very strong pre-processing filter and inducting confidence scores is not needed. we also determined that monolingual language models can help us in selecting sentences even if both the source and target language models are independent of each other. finally, using using the length of a sentence as a filter helps us create a better nmt system. we also learn that statistical intuitions do not easily extend to neural embeddings. ## baseline the idea behind our baseline system is to use the cosine distance between two multilingual representations as a notion of parallelism between sentences embedded in the same space. the tool we use for this is laser foot_0 which uses an encoder-decoder architecture to train a multilingual sentence representation model. it has been shown by @xcite that laser is effective at zero-shot cross-lingual natural language inference in the xnli dataset which makes it promising for this task involving low-resource languages. @xcite has shown this to be a strong baseline. we follow the work done by @xcite and begin by generating multilingual sentence embeddings using laser. the laser score is a function of the margin between the cosine similarity between a given candidate and it's k nearest neighbors foot_1 : let f(x,y) = z n n k (x) cos(x, z) 2k + z n n k (y) ## language id for our first step, we try to predict the most probable language of a given sentence using use fasttext @xcite . we use the pre-trained model released by the authors that is trained to identify over 170 languages including pashto, khmer and english. the intuition behind it is that when working in a bilingual setting, sentences from other languages or code-mixed sentences will be harmful to the mt system. we call this simple language id if @xmath0. for example, if we have a sentence where the probability of the target sentence being english is 0.9 and we have another sentence where the probability of the target sentence being english is 0.3, then given the same laser scores, we would want to give preference to the sentence pair where the probability of the target sentence is 0.9. we do this for both the source and target language. we try to only keep sentence pairs where we are confident about both the source and target language being correct. we implement this notion by setting a cutoff c for the language id probability. we call this confident language id if prob(@xmath1. we show our results for pashto in we experiment by including scores if they are within the top 3 predicted languages but see no significant change in scores. we also experiment by adding the confidence instead of multiplying it in confident language id but see no significant change in the bleu scores. as a result of our observations, we only perform simple language id for khmer, giving us a bleu score of 11.51 for pashto and 10.04 for khmer. we interpret the norm of an embedding as a measure of the confidence we have in the embedding of a sentence. the reason we do this is that neural methods by their very nature are data hungry and susceptible to noise. we are working in a low-resource condition because of which it will be harder to learn about sentences with contextsensitive words. additionally, it would be harder to learn about sentences with low-frequency words making their embeddings less reliable, thus, leading to a lower quality mt system. we run 2 sets of experiments on the laser embeddings of the sentences. the langid scores we have until now are in the range [0,1], while the norm of the embeddings theoretically have a range of [0, ∞). to ensemble the langid scores with the norm-scores, we need to bring the distributions within a comparable range. we do this by applying min-max normalization. let @xmath2, x 2 , ..., x n ] where we finally ran experiments where we gave a preference to only large norms and an experiment where we gave a preference to both very large norms and to very small norms. in both the cases we had really bad scores leading to the conclusion that the norm of laser embeddings is not a good filter. because of this poor performance when relied on aggressively, we decided not to use it in our final submission. we also ran the same experiments using fasttext generated embeddings @xcite but that had poor results as well. ## monolingual language models the motivation behind using monolingual language models is that we want to learn about sentence pairs that have a high likelihood of coming up in the test data. ideally we would want both the sides of the corpus to have a high probability of coming up but we also realize it will often not be the case. thus, we make these language models independent of each other. we take inspiration from @xcite and modify the work of junczys-dowmunt (2018) to come up with a language model filter. junczys-dowmunt (2018) achieved the highest ranking score in wmt'18 and to do so they define h m (.|.) as the wordnormalized conditional cross-entropy of the probability distribution p m (.|.) for a model m: we use this as a measure of how fluent a given sentence is. lower scores indicate a better sentence in this case. while @xcite do create n-gram language models, they hope that language models trained on similar but not parallel texts to have similar perplexities over each half of a parallel test set of the parallel corpus. this method does not leverage the simple fact that we have more data for one of the languages. this method also assumes a close relationship between the frequencies of letters which might not always be the case. finally it does not leverage the expressive power of neural language models. in order to improve on this, we propose neural language models that are completely independent of each other. ## final submission our final pipeline is as follows: we submit what we believe to be the 3 most robust solutions we have for each language pair. for pashto, we apply the language id filter, then we use the transformer architecture language model, set lr to 3 and set(α, lc) to (0.8, 5), (0.9, 0) and (0.9, 5). for khmer, we apply the language id filter, then we use the transformer architecture language model and set (α, lc) at (0.8, 5) and (0.8, 6). apart from that, we also submit a scores with a filter checking for token overlap over 30%. the (α, lc) is set to (0.8, 5). at the time of submission our best score for pashto is 11.69 and the best score for khmer is 10.24 on the development set. the findings of the shared task presented by @xcite