x SIM ++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages (2023)

Abstract: We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xsim++. In comparison to xsim, this improved proxy leverages rule-based approaches to extend English sentences in any evaluation set with synthetic, hard-to-distinguish examples which more closely mirror the scenarios we encounter during large-scale mining. We validate this proxy by running a significant number of bitext mining experiments for a set of low-resource languages, and subsequently train NMT systems on the mined data. In comparison to xsim, we show that xsim++ is better correlated with the downstream BLEU scores of translation systems trained on mined bitexts, providing a reliable proxy of bitext mining performance without needing to run expensive bitext mining pipelines. xsim++ also reports performance for different error types, offering more fine-grained feedbacks for model development.

## introduction when training neural machine translation (nmt) systems, it has been shown in prior works that generally, the quality of such systems increases with the availability of high-quality training data @xcite . however, for many low-resource languages there are few public corpora available, posing many challenges. in order to address this sparsity, one approach is to supplement existing datasets with automatically created parallel corpora, and a technique which has shown to be successful for such issues is the task of bitext mining @xcite . in bitext mining, the aim is to find pairs of sentences with the same sentence meaning across collections of monolingual corpora. in this work, we adopt a global mining approach @xcite , which has shown recent success in provid- * equal contribution ing high-quality data for low-resourced languages @xcite . in order to evaluate any bitext mining method, a natural approach is to train a nmt system on the automatically created alignments. however, this is extremely costly. as an alternative, the bucc task @xcite offers a method for evaluating bitext mining algorithms by embedding known alignments within monolingual corpora, and then reporting on the number of correctly aligned pairs. however, this task currently only covers 5 high-resourced languages (english, french, russian, german and chinese), and so is not applicable to the low-resource domain. in order to address this, another approach to evaluate bitext mining is to align existing multilingual parallel test sets. two such test sets are tatoeba foot_0 and flores200. 2 however, as shown by @xcite , the tatoeba corpus is not very reliable given that for some sentence pairs there are only a few hundred sentences. therefore, we opt to use flores200, which is also n-way parallel. one existing method for evaluating bitext mining on parallel test sets is xsim. 3 this method reports the error rate of misaligned sentences, and follows a margin-based global mining approach @xcite . however, although using xsim on test sets such as flores200 has been shown to be useful as a proxy metric for bitext mining @xcite , it has the following limitations: transformation category original sentence transformed sentence causality alternation apart from the fever and a sore throat, i feel well and in good shape to carry out my work by telecommuting. apart from the fever and a sore throat, i feel well and in bad shape to carry out my work by telecommuting entity replacement charles was the first member of the british royal family to be awarded a degree. m. smith was the first member of the university to be awarded a degree. number replacement nadal bagged 88% net points in the match winning 76 points in the first serve. nadal bagged 98% net points in the match winning 71 points in the sixth serve. in order to address these limitations, in this work we introduce xsim++. this is an improved proxy for bitext mining performance which expands the dev and devtest sets of flores200 to include both more data points, and also difficult to distinguish cases which provide far greater challenges to the models. our contributions can be summarised as follows: ## experiment in order to establish xsim++ as a proxy for bitext mining performance, we measure the correlation between both xsim and xsim++ error rates, and the bleu scores resulting from nmt systems trained on mined bitexts. more specifically, for each language we choose a sentence encoder model, followed by bitext mining using each respective encoder, and then train and evaluate bilingual nmt systems on the resulting mined bitexts. we use the flores200 development sets when computing the bleu scores. in order to validate xsim++ against varied embedding spaces, we encode (and mine) using two different multilingual encoder methods: laser @xcite and labse @xcite . for laser, we trained our own custom encoders (details below). for labse, we used a publicly available model foot_7 as the code and data for training labse are not publicly available. we randomly choose 10 low-resource languages to perform both encoder training (if applicable) and bitext mining. the languages are: faroese (fao), kabuverdianu (kea), tok pisin (tpi), kikuyu (kik), friulian (fur), igbo (ibo), luxembourgish (ltz), swahili (swh), zulu (zul), bemba (bem). encoder training. we trained laser encoders using the teacher-student approach described in @xcite . we choose a laser model @xcite as our teacher, and then trained specialised students for each language. in order to train each student, we used both publicly available code foot_8 and bitexts (e.g. opus foot_9 ) bitext mining. for each chosen encoder model, we perform bitext mining against approximately 3.7 billion sentences of english. for low-resource languages, the sizes of monolingual data range from 140k to 124 million. details are in the appendix. we make use of monolingual data available from both commoncrawl and paracrawl foot_10 , and operationalize the mining using the stopes library @xcite . 12 for laser, we use 1.06 as the margin threshold following @xcite and for labse, we use 1.16. 13 following mining, for each language we concatenate publicly available bitexts and the mined bitext as training data for nmt bilingual models using fairseq, foot_13 translating from each foreign text into english. for all nmt systems, we keep the hyperparameters fixed (details in appendix). evaluation. model selection involves two use cases: comparisons within a model and across different models. for the former comparison, given our custom encoders, we choose to compare 10 checkpoints from each model. 15 for cross model comparisons, we compare each chosen encoder checkpoint against another existing system. in this case, the labse encoder. to quantitatively measure these two cases, we report pairwise ranking accuracy @xcite for xsim and xsim++. formally, the accuracy is computed as follows ## related work as xsim++ uses rule-based data augmentation, it is related to work in other areas that also employ similar data augmentation methods, such as part-ofspeech tagging ( şahin and steedman, 2018), contrastive learning @xcite ), text classification @xcite , dialogue generation @xcite and summarization (chen and yang, 2021). ## conclusion and future work we proposed a proxy score xsim++ for bitext mining performance using three kinds of data augmentation techniques: causality alternation, entity replacement, and number replacement. to validate its effectiveness, we conducted large-scale bitext mining experiments for 10 low-resource languages, and reported pairwise ranking accuracies. we found that xsim++ significantly improves over xsim, doubling the accuracies. analysis reveals that entity replacement helps most in the improvement. in future work, we plan to extend xsim++ to non-english language pairs.