Quality Estimation of Machine Translated Texts based on Direct Evidence Approach (2024)

Abstract: Quality Estimation task deals with the estimation of quality of translations produced by a Machine Translation system without depending on Reference Translations. A number of approaches have been suggested over the years. In this paper we show that the parallel corpus used as training data for training the MT system holds direct clues for estimating the quality of translations produced by the MT system. Our experiments show that this simple, direct and computationally efficient method holds promise for quality estimation of translations produced by any purely data driven machine translation system.

## introduction the performance of machine translation (mt) systems is measured either using manual evaluation, using metrics such as adequacy and comprehensibility, or using automatic methods, using metrics such as bleu and ter, by comparing with @xcite quality estimation (qe), on the other hand, deals with automatic estimation of quality of translations produced by an mt system without using reference translations. machine translation generally works sentence by sentence and the primary goal of the quality estimation task is also to measure the quality of translations at sentence level. qe of mt outputs has several benefits. good translations can be selected, postedited as required and added to the training data. poor quality instances can be removed from training data to reduce noise. qe helps in more accurate estimation of post-editing time and effort and in taking associated decisions in commercial translation. parallel corpus data required for building mt systems is generated today mostly using automatic methods such as web crawling and back translation, and the data so generated is usually quite noisy. modern deep neural architectures are data hungry. quantity and quality of data are thus both very important. neural mt (nmt) performance, for example, has been shown to be highly dependent on the size of the training data @xcite as well as the quality @xcite . a large number of techniques have been proposed for quality estimation. the annual workshop on machine translation (wmt) has been including a shared task on quality estimation for many years now. a typical approach is to use manually scored translations as training data for training a machine learning system to perform the qe task. more recently, sentence embeddings generated using deep learning neural network architectures such as transformer, have been used to estimate the quality of translations. the core idea is to compare the sentence embeddings of the source language sentence with that of the target language sentence. higher the similarity of sentence embeddings, higher is the semantic similarity of the sentences. language agnostic bert sentence embedding model (labse) @xcite is an example of this approach. in this paper, we propose a much simpler and more direct approach called the direct evidence approach. we show that while being computationally much less expensive, our method correlates well with other more sophisticated methods such as labse. while the research in mtqe is rich in terms of ideas, techniques, tools and resources, it appears that none of them are directly looking at the parallel corpus that is used for building mt systems, for clues about quality of translations. here we propose what we call direct evidence approach, which is based directly and solely on the training data that is used to build mt systems. no other training data or resource is required. this approach is also conceptually simple and computationally very efficient. ## related work till recently, the mainstream approach to qe was to use a set of manually scored translations to train a machine learning system. most of the works reported in the wmt shared tasks on qe used this approach. several sub-tasks and related tasks were also taken up in the wmt workshops. word level qe deals with marking of words as ok or bad. in fact, sentence level scores were often computed or estimated using these word level scores. scoring entire documents was another task. identifying source language (sl) words that cause quality issues was also looked at. explainable qe task and critical error detection task were included in the wmt-2022 conference. both direct assessment on post-edit data (called mlqe-pe) and multidimensional quality metrics (mqm) were included. in the prevalent evaluation practices, qe systems were assessed mainly in terms of their correlation with human judgements. zerva et al. (2022) describe the findings of the 11th edition of the qe shared task held as part of wmt-2022. participants from 11 different teams submitted altogether 991 systems to dif-ferent task variants and language pairs in wmt-2022. zaretskaya et al. (2020) ask whether the current qe systems are useful for mt model selection. gladkoff et al. (2022) focus on the amount of data that is required to reliably estimate the quality of mt outputs. they use bernoulli statistical distribution modeling and monte carlo sampling analysis towards this end. don-yehiya et al. ( 2022 ) focus on quality estimation of machine translation outputs in advance. they present a new task named prequel, the task of predicting the quality of the output of mt systems based on the source sentence only. @xcite summarize the findings of the 2023 edition of the wmt conference. khayrallah and @xcite explore how various types of noise in the training data impact the quality of neural machine translation systems. they find that neural models are generally affected more by noise than statistical models. nmt performance degraded by 9.9 bleu points when noise was added while statistical machine translation (smt) actually gained 1.2 bleu points. effect of various types of noise such as misaligned sentences, misordered words, wrong language, untranslated or very short segments and raw crawl data have been explored. @xcite introduce language agnostic bert sentence embedding (labse), and compare with laser (language agnostic sentence embedding representations) @xcite and m-use (multilingual unsupervised and supervised embeddings) @xcite approaches. labse outperformed laser and m-use in many scenarios. labse is a multilingual sentence embedding model for more than 109 languages based on dual encoder transformer architecture of @xcite ; @xcite . all the above three models are computationally highly expensive. for example, labse uses the bert base encoder architecture with 12 transformer blocks, 12 attention heads, 768 per-attention hidden units. sentence embeddings are extracted as l2 normalized [cls] token representation from the last transformer block. models are trained on cloud tpu v3 with 32-cores using a global batch size of 4096, with a maximum sequence length of 128, using adamw optimizer with initial learning rate e -3 and linear weight decay. the default margin value for additive margin softmax is set to 0.3. @xcite introduce a few-shot transfer learning based approach to qe and show that using this approach for corpus filtering gives higher improvements in mt performance compared to labse based corpus filtering. @xcite explore various data filtering methods and evaluate them on the downstream task of nmt. they conclude that cross entropy based filtering outperforms other approaches. @xcite view corpus refinement as an outlier detection task. in order to detect and remove the mistranslations in a parallel corpus, they map each sentence pair into an n-dimensional feature space and then estimate the density for each one of them. the least dense points are treated as outliers and are removed from the corpus. @xcite present indictrans2, claiming to provide high quality and accessible mt models for all the 22 scheduled indian languages. they release bpcc, a parallel corpus including a total of 230 m bitext pairs, of which about 126 m were newly added in this release, including 644 k manually translated sentence pairs. they also released the first n-way parallel benchmark covering all 22 indian languages. @xcite aim to remove the incorrect translations from the dataset to make the translation quality better. sentences with poor translation quality (bleu score lesser than a threshold) are treated as noise and discarded from the dataset. @xcite propose a novel approach to filter this noise from synthetic data. for each sentence pair of the synthetic data, they compute a semantic similarity score using bilingual word embeddings. @xcite propose a fast and scalable data cleaning system for noisy web-crawled parallel corpora. they propose a novel type of bagof-words translation features, and train logistic regression models to classify good data and synthetic noisy data in the feature space. @xcite introduce opusfilter, a flexible and modular toolbox for filtering parallel corpora. in contrast to tools such as bicleaner @xcite and zipporah xu and koehn (2017 ) that implement a single method for parallel corpus filtering, opusfilter is designed as a toolbox that is useful for testing and using many different approaches. @xcite propose a graph based random walk approach to clean bilingual data for smt. a pagerankstyle random walk algorithm brin and page (1998); @xcite ; @xcite is used to iteratively compute the importance score of each sentence pair that indicates its quality: the higher the better. unlike other data filtering methods, their proposed method utilizes the importance scores of sentence pairs as fractional counts to calculate the phrase translation probabilities based on maximum likelihood estimation. @xcite present the findings of the wmt 2023 shared task on parallel data curation. they pose the open-ended shared task of finding the best subset of possible training data from a collection of estonian-lithuanian web data. junczys-dowmunt (2018) introduce dual conditional crossentropy filtering for noisy parallel data. for each sentence pair of the noisy parallel corpus they compute cross-entropy scores according to two inverse translation models trained on clean data. they penalize divergent cross-entropies and weigh the penalty by the cross-entropy average of both models. sorting or thresholding according to these scores results in better subsets of parallel data. @xcite explore how prompt strategies affect downstream translation performance. then, they conduct extensive experiments with two finetuning methods, three llm backbones and 18 translation tasks across nine language pairs. their findings indicate that in some cases, these specialized models even surpass gpt-4 in translation performance, while they still significantly suffer often from offtarget translation issue, even if they are exclusively fine-tuned on bilingual parallel documents. ## direct evidence approach translation is a meaning preserving transformation of texts from a source language (sl) to a target language (tl). this is generally done sentence by sentence, or more generally, segment by segment. in order to preserve the meaning of the sl sentence, words and phrases in sl sentences need to be mapped to equivalent words and phrases in the tl. other aspects of syntax and semantics such as agreement, word order, semantic compatibility will also need to be addressed. modern purely data driven approaches such as statistical machine translation and neural machine translation are based on the view that all linguistic regularities and idiosyncrasies are indirectly present in the parallel corpus and parallel corpus alone is sufficient, no other data or linguistic resource is needed. a machine translation system can be obtained by training on a training data set consisting of a parallel corpus alone. we believe that the training data also has clues useful for estimating the quality of translations produced by the mt system. in particular, here we focus on lexical transfer. we show that the word co-occurrence matrix (wcm) holds direct clues for estimating the quality of lexical transfer and hence quality of translation as a whole. statistical basis for performing lexical transfer comes mainly from word co-occurrence statistics. let sl-tl be a parallel corpus consisting of n source language segments s 1 , s 2 , s 3 , ..., s n , paired with their translational equivalents t 1 , t 2 , t 3 , ..., t n in the target language. we say sl word i co-occurs with tl word j if the tl word j occurs anywhere in the translational equivalent of a sl sentence in which the word i occurs. let v s be the vocabulary of the source language (total number of distinct word forms occurring in any of the sl segments) and v t be the vocabulary of the target language. then word co-occurrence matrix wcm is a v s x v t matrix of non-negative integers where w cm i,j indicates the total number of times the source language word i had co-occurred with the target language word j in the entire training data set. clearly, wcm will be a very large and very sparse matrix. a large w cm i,j value indicates a strong correlation between the sl word i and tl word j in the training corpus. if an sl word i co-occurs with a tl word j large number of times, if i does not occur with too many other tl words with high frequency, if the wcm counts for other possible mappings in tl are significantly lower, all these indicate that the lexical transfer of i to j during translation can be done with high confidence. when the evidence in the form of co-occurrence counts coming from the training data is weak, the mt system may still go ahead and substitute the word j for word i based on the combined evidence coming from other parts of the sentence, language model, etc. this may be an optimal decision taken by the mt system with regard to some specified loss function. optimal choice in some probabilistic sense may not be the correct choice, it may just be the best of several possible choices, none of which may be correct. mt systems generally go ahead and produce translations, whether they are sure or not-so-sure or notat-all-sure. here we hypothesize that the fraction of words in a sl sentence that have strong cooccurrence relations with any of the words in the tl sentence produced by the mt system, is a direct indicator of quality of translation. we call this direct evidence approach. ## methodology first we compute the word co-occurrence matrix wcm from the mt training data. then for each sl-tl sentence pair in the test set, we check the number of sl words (excluding stop words) for which there is 'strong evidence' in the training data. a source language word w is said to have 'strong evidence' if it co-occurs at least t times with any of the target language words in the sentence pair, where t is a specified threshold. we call the percentage of sl words with 'strong evidence' as the de score for the sentence pair. de scores range from 0 to 100. de score is taken as a measure of quality of translation. ## conclusions in this paper we hypothesize that the parallel corpus used for training an mt system holds clues about the quality of translations the mt system can produce. we propose a simple and direct approach to quality estimation based solely on the training data. a word co-occurrence matrix is constructed from the training corpus and used to estimate the sentence by sentence quality of translations. each sentence gets a score called de score, which is indicative of the quality of translation. manual observations show that good quality translations generally tend to get higher de scores and poor quality translations tend to get lower scores. our experiments reconfirm this. this simple and direct evidence approach to mt quality estimation appears to holds promise. we can estimate the quality of translations even without / before running the mt system. we do not need any other data or resource, we only need the training corpus. de-scores provide us a spectrum of quality grades and since they are based on co-occurrence counts, out of vocabulary (oov) words are only cases that lie just outside the low end of this spectrum. missing words automatically get reflected in poor de scores but extra words in tl can be detected by performing a tl to sl wcm check. if large scale manual post-edit data such as hter scores are available, then we can estimate the various thresholds using machine learning techniques instead of using human judgement as we have done here. in summary, de score is a simple, direct and efficient method for the qe task. we can also easily deal with untranslated text, third language, emojis and other invalid characters etc.