Better Quality Estimation for Low Resource Corpus Mining (2022)

Abstract: Quality Estimation (QE) models have the potential to change how we evaluate and maybe even train machine translation models. However, these models still lack the robustness to achieve general adoption. We show that Stateof-the-art QE models, when tested in a Parallel Corpus Mining (PCM) setting, perform unexpectedly bad due to a lack of robustness to out-of-domain examples. We propose a combination of multitask training, data augmentation and contrastive learning to achieve better and more robust QE performance. We show that our method improves QE performance significantly in the MLQE challenge and the robustness of QE models when tested in the Parallel Corpus Mining setup. We increase the accuracy in PCM by more than 0.80, making it on par with state-of-the-art PCM methods that use millions of sentence pairs to train their models. In comparison, we use a thousand times less data, 7K parallel sentences in total, and propose a novel low resource PCM method.

## introduction the quality estimation (qe) task aims to model human perception of translation quality and predict the quality score an expert would give to a translation using only the source sentence and the translation. this requires the qe model to represent the cross-lingual similarity between source and hypothesis sentences while incorporating different features of the hypothesis sentence such as fluency, grammaticality and adequacy foot_0 . human evaluations of machine translation are costly and time-consuming for a large-scale text dataset. references to evaluate machine translation performance are not readily available in many cases, especially in low-resource languages. even if they do exist, they often assume a single, unique answer for correct translations, causing bias in the evaluation. thus, it is academically and professionally of paramount importance to further develop reliable quality estimation metrics, which can ultimately eliminate the need for references and have unlimited potential for practical applications in machine translations. parallel corpus mining (pcm) is another critical task that can enable the creation of high-quality parallel data and reduce the need for considerable human effort. these mined parallel corpora could especially be helpful in low resource languages. on the other hand, current pcm methods require large amounts of parallel data, which creates a paradoxical loop that only large companies can break. quality estimation is uniquely linked with pcm since what makes a good translation most of the time makes a correct parallel too. considering the similarity in the underlying goals of these two tasks, we expect models that can do one to perform, at least, acceptably in the other. however, @xcite have shown that models that can do corpus mining fail in qe and propose a resource prudent method to bridge the gap. we show that the gap exists in the other direction and we introduce simple and valuable solutions. in chapters 3 and 4, we introduce our method multiqe and its base variants. since we do not want to depend on additional cross-lingual data, we propose using multitask training with monolingual linguistic inference and semantic similarity data. we also experiment with using multitask feature extraction and compare our methods with sota qe methods in the multilingual quality estimate (mlqe) dataset @xcite . in chapter 5, we use data augmentation techniques in combination with multitask training to train more robust qe models and check their robustness in the parallel corpus mining setup using the tatoeba @xcite and bucc @xcite datasets. we use the term robust qe models to refer to models that can overcome the problem of just focusing on grammaticality/fluency, which causes sota qe models to fail in pcm. our method outperforms sota qe models on pcm with a substantial margin, up to 0.80 difference in accuracy score in tatoeba. in addition, we compare our method with high resource methods like laser @xcite and labse @xcite which are trained on vast amounts of parallel data and achieve sota performances on pcm. our method essentially offers a better and more robust qe model that is trained with very little data (thousand times less data) compared to these models. the goal in comparing to these high resource methods is to show that our proposed method achieves good enough performance to be a viable low resource pcm method. our contributions in this paper can be summarized as below. ## quality estimation experiments this section will go over the dataset, results, significance test, and ablation study for our experiments on the mlqe dataset. ## parallel corpus mining experiments in the pcm experiments, we will use multiqe multitask because it performs the best in ## discussion we show that semantic similarity and linguistic inference improve qe performance. we test for significance and show that our methods outperform sota qe methods( this intuition that pretraining with related tasks, especially with sts and nli, is helpful for evaluating translations is in line with background and findings from @xcite and @xcite . moreover, qe benefiting from monolingual data shows that xlm-r can utilize the labels in monolingual datasets to make better inferences in a cross-lingual task. this is most likely because it is already a cross-lingual language model. additionally, we show that multitask training for qe can improve the robustness of the model. we demonstrate improvements in accuracy around 0.50 in the tatoeba experiments ( we show that sota qe models yield unexpectedly poor performance in a pcm setting( our qe models process input sentences as pairs, bringing up the computational cost problem. solving this with the sentence filtration model we train using contrastive loss enables the use of our qe method in large filtration tasks. making it a good low resource corpus mining method that can achieve on par results with sota methods (tables 4 and 5 ). the importance of this contribution is amplified when we consider that our method is trained only using 7k parallel sentences compared to other pcm methods, which are trained on the order of millions of sentences. throughout our experiments, we keep low resource limitations in mind. while we acknowledge that collecting more data across different families of languages is an option to scale methods to low resource languages. we argue that exploring the improved usage of less data with better labels promises another important avenue to make useful methods like qe or pcm available in low resource languages. ## future work to further our work, we plan to explore contrastive loss fine-tuning with self-supervision to improve qe performance planning and further reduce the need for labels. self-supervised learning is an exciting way of forcing a neural language evaluator to abstain from certain mistakes. this approach can force invariance or target to reduce certain types of errors. the nature of the information attained by the network is primarily dependent on the negative sample generation process. another interesting avenue to explore is using qe in active learning for machine translation as a scheduling or training signal.