“A Little is Enough”: Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation (2023)

Abstract: Quality Estimation (QE) is the task of evaluating the quality of a translation when reference translation is not available. The goal of QE aligns with the task of corpus filtering, where we assign the quality score to the sentence pairs present in the pseudo-parallel corpus. We propose a Quality Estimation based Filtering approach to extract high-quality parallel data from the pseudo-parallel corpus. To the best of our knowledge, this is a novel adaptation of QE framework to extracting quality parallel corpus from the pseudo-parallel corpus.. By training with this filtered corpus, we observe an improvement in the Machine Translation (MT) system’s performance by up to 1.8 BLEU points, for English-Marathi, Chinese-English, and Hindi-Bengali language pairs, over the baseline model. The baseline model is the one that is trained on the whole pseudo-parallel corpus. Our Few-shot QE model transfer learned from the English-Marathi QE model and fine-tuned on only 500 Hindi-Bengali training instances, shows an improvement of up to 0.6 BLEU points for Hindi-Bengali language pair, compared to the baseline model. This demonstrates the promise of transfer learning in the setting under discussion. QE systems typically require in the order of (7K-25K) of training data. Our Hindi-Bengali QE is trained on only 500 instances of training that is 1/40th of the normal requirement and achieves comparable performance. All the scripts and datasets utilized in this study will be publicly available.

## introduction in recent times, neural mt has shown excellent performance, having been trained on a large amount of parallel corpora @xcite . however, not all language pairs have a substantial amount of parallel data. hence, we have to rely on the noisy web-crawled corpora for low-resource languages. the task of parallel corpus filtering aims to provide a scoring mechanism that helps extract goodquality parallel corpus from a noisy pseudo-parallel corpus. the task of quality estimation (qe) aims to provide a quality score for a translation when the reference translation is unavailable. we use quality estimation to assign the quality scores to the sentence pairs present in pseudo-parallel corpora and extract good-quality parallel sentences. we aim to improve the quality of machine translation for english(en)-marathi(mr), hindi(hi)-bengali(bn) and chinese(zh)-english(en) language pairs by using sentence-level qe-based corpus filtering. we observe that qe-based corpus filtering performs better than previously proposed methods. our contributions are: ## approaches we first discuss methods to extract good-quality parallel sentences from the pseudo-parallel corpus. then we discuss a transfer learning-based filtering approach in few-shot settings. ## mathematical preliminaries labse scoring let @xmath0, y i )} n @xmath1, where x i and y i represents i th source and target sentence respectively. we first feed all the source sentences present in the pseudo parallel corpus as input to the labse foot_3 @xcite ) model, which is a dual encoder model with bert-based encoding modules to obtain source sentence embeddings (s i ). the sentence embeddings are extracted as the l2 normalized [cls] token representations from the last transformer block. then, we feed all the target sentences as input to the labse model to obtain target sentence embeddings (t i ). we then compute cosine similarity (score i ) between the source and the corresponding target sentence embeddings. parallel corpus consists of high-quality sentence pairs, while the pseudo-parallel corpus contains sentence pairs of varying quality. the en-mr parallel corpus consists of the ilci phase 1, bible, pib and pm-india corpus @xcite @xcite . the zh-en parallel corpus consists of paramed foot_4 corpus. the hi-bn parallel corpus is obtained from the opus foot_5 corpus repository. the en-mr and zh-en pseudo-parallel corpus consist of the samanantar @xcite and wmt18 zh-en foot_6 corpus, respectively. the hi-bn pseudo-parallel corpus consists of samanantar and tatoeba (tiedemann, 2020) corpus. the detailed data statistics are mentioned in in qe experiments, we create a small corpus (500 instances) for hindi-bengali language pair that consists of human-annotated domain adaptation scores for each sentence pair annotated by three annotators. the pairwise pearson correlation between the three annotators of hindi-bengali qe is 0.68, 0.61 and 0.67. this indicates a good agreement between the three annotators. please refer to appendix a.3 for further annotation details. we use the qe data provided by @xcite and @xcite for the zh-en and en-mr language pairs, respectively. the detailed qe data statistics are mentioned in for evaluation, we use the flores 101 test set which contains 1,012 sentence pairs for each language pair. ## results and analysis we evaluate our nmt models using bleu @xcite . we use sacrebleu @xcite python library to calculate the bleu scores. ## conclusion and future work we introduced a simple quality estimation based corpus filtering approach to extract high-quality parallel data from the noisy pseudo-parallel corpora. the takeaway from our work is that sentence-level qe-based filtering performs better than labse-based filtering and helps improve the performance of nmt systems. we also show that few-shot qe models trained using a transfer learning-based approach can be used to extract good-quality parallel corpus from the pseudoparallel corpus. only 1/40 th of the normal data requirement (7k-25k) of qe training data achieves comparable performance for the hindi-bengali language pair. we also show that the qe quality score is superior to the labse quality score. in the future, we plan to use the proposed corpus filtering technique for other language pairs. this will provide us with a general overview of how this filtering technique performs for multiple languages.