All You Need is Source! A Study on Source-based Quality Estimation for Neural Machine Translation (2022)

Abstract: Segment-level Quality Estimation (QE) is an increasingly sought-after task in the Machine Translation (MT) industry. In recent years, it has experienced an impressive evolution not only thanks to the implementation of supervised models using source and hypothesis information, but also through the usage of MT probabilities. This work presents a different approach to QE where only the source segment and the Neural MT (NMT) training data are needed, making possible an approximation to translation quality before inference. Our work is based on the idea that NMT quality at a segment level depends on the similarity degree between the source segment to be translated and the engine’s training data. The features proposed measuring this aspect of data achieve competitive correlations with MT metrics and human judgment and prove to be advantageous for post-editing (PE) prioritization task with domain adapted engines.

## introduction quality of neural machine translation (nmt) systems keeps improving and gives humans the ability to translate enormous amounts of segments in a short time. however, raw machine translation is seldom perfect. therefore, mt in standard localization processes is most of the times followed by some level of human or automated editing aimed at fixing issues in the mt output. in the translation industry we are witnessing a surge in demand for translation services, as well as increased requests for raw mt (without human review). often, clients are very concerned about their translation spend, or they do not have time to translate all the content they would like to see translated, therefore more and more of them look for raw machine translation services to get savings and quicker turnaround time. however, depending on the language pairs, use cases and content types involved, raw machine translation for direct consumption (without pe) might not be a good solution. in an ideal scenario, the quality of the output delivered by mt engines is measured before they are used in production. this exercise is aimed to understand if mt will be helpful for the linguist, or even just to understand if a mt engine training was successful or not. typically, the quality of mt translations is measured by comparing how different the mt output is from its reference translation. but how do we measure the quality of mt if we do not have a reference translation? it happens very frequently: imagine that you need to translate a new content type or into a new language pair for which you do not have any reference translation. this conflict led to the recent emergence of quality estimation (qe) techniques that try to estimate the quality of a translation when the reference information is not available. in wmt20 qe shared task, state-of-theart (sota) qe models were supervised models trained exclusively on labeled data composed of source segments, the corresponding translations, and human direct assessment (da) (wmt20 qe findings, 2020). the same year, an unsupervised method was proposed to estimate translations @xcite . the paper intro-duced the idea of using nmt as a glass-box to estimate translation quality by using token level probabilities. from that breakthrough, wmt21 sota qe models combined the supervised and unsupervised approaches (wmt21 qe findings, 2021). despite the outstanding results of these qe models, we observed that they cannot easily be implemented in production environments for different reasons. firstly, because a substantial amount of human-labeled data is needed to fine-tune such architectures for a specific language pair and domain. in many cases, it can be problematic to find or generate this type of data without creating some domain shift between the qe training data and the data that has to be estimated in production. this can lead to catastrophic results. secondly, these models rely on large language models that are expensive to train, store and run. thirdly, depending on how the adapted nmt models are put in production, they can deprive the owner of nmt probabilities used as features which can also be computationally expensive to extract. this work tries to elude these challenges by proposing a more "data-centric" direction to estimate nmt quality. indeed, the importance of data in nmt has been extensively studied in different fields such as domain shift @xcite , catastrophical forgetting @xcite and domain robustness @xcite . hence, it is well known that adapted models will have higher performance on segments from the same domain, or similar to the ones contained in the training data in some aspect. in this perspective, we think that there could be a way to estimate the nmt performance on a segment by checking the source segment and comparing it to the source segments contained in the training data. this work presents two simple techniques to perform this task: a) by measuring the similarity between the segment to be translated and the source segments found in the training data and b) by counting the number of words in the source segment that do not appear in the engine training data (unknown words for the engine). we evaluate our approach in two steps. firstly, we create generic engines in three language pairs, and then we adapt each one of them with clientspecific data. with these six engines, we translate a set of segments for which the reference is known, score at a segment level those translations using bleu, chrf3 and comet, and compute our new source-similarity features. after that, we study and discuss the correlation between these new features and the segment-level mt metrics and human evaluations. secondly, we focus on the in-domain scenarios to evaluate the impact of using this simple approach as qe metrics to prioritize the segments to be post-edited. our main contributions at the end of this study are: (a) a simple, unsupervised and effective approach to estimate the mt quality without checking the reference translation or before producing the translation; (b) an evaluation of how these features correlate with several mt scores and human judgement, both in generic and adapted nmt systems, similarly to previous qe methods; (c) an evaluation of how these features can be used as competitive indicators to prioritize segments to be post-edited. while the study focuses on an unsupervised segment level usage, it opens the door to explain quality changes at a project level and can inspire future architectures for qe models where the source side similarity information could be included. ## source qe for nmt in this work, we try to extract information that can describe how familiar a segment is to a given engine by comparing each source segment that needs to be translated against all the source segments included in the training data. the two following subsections propose features by transforming the segments into vectors and getting some statistical measurements of the vectors' similarity. these vector similarities are computed with the cosine similarity defined as follows for vectors a and b: with this score we can capture how similar the vectors are. in absolute terms, the values returned are contained in [0,1]: values approaching 1 represent high similarity, while values closer to 0 represent low similarity. for explanatory purposes, we denote s train the set of n train source segments composing the training data and s test the set of n test source segments to translate. ## first experiment and results in the following experiment, we describe correlations between the proposed features and the previous mt metrics for generic and domain-adapted systems trained as explained in section 4. additionally, we compute the correlations with da for the in-domain translations with data described in the same section. note that, to compute the indicators for the adapted engine, only the in-domain training data is considered to compare the source segments. ## business implementations there are many scenarios in which this feature could be useful in production, for a language service provider. while we briefly mentioned quite a few ideas in this paper, we would now like to focus on the implementation that we believe would bring the biggest benefit to the client. if we used unk+max sem to identify a fixed amount of mostly challenging segments, by looking at the source only, and decided to post-edit only this sample of potentially incorrect segments, the client could get a dramatic improvement in the quality of the content translated with a little effort. by knowing the budget of the client for translating a document, we could estimate the number of words that can be post-edited with that budget and the extent of the improvement we could get. let's assume that the client has budget (or time) only to post-edit the 10% of the document. in a traditional scenario, the client would probably rather have raw mt on everything, prioritize post-editing only on those part of the content (if any) that get more visibility, or even worse, post-edit only some randomly selected chunks of text. conversely, by using these indicators, we could aim at performing post-editing only on the top 10% segments that we know have a higher probability of containing issues. similarly, if the client has no fixed budget or turnaround time, but is trying anyway to save as much money and time as possible, we could recommend that they do pe only on that percentage of segments which could increase bleu. this estimation would help them publish their content more quickly, because part of it would not need any human intervention and would enable linguists to focus only on what really needs to be fixed. also, while there might of course still be errors in the mt output that do not get reviewed, this approach gives clients with budgetary constraints a focused way to spend and some certainty that the worst segments will not reach the reader. ## conclusions in this paper, we offered a new approach to unsupervised segment-level qe for nmt systems by only evaluating the source segment with the help of nmt training data. by using sentence transformers and bag of words methods, we transformed all the segments into vectors and computed the maximum semantic and string similarity. these scores, along with a feature counting the number of unknown words for the nmt system, seem to contain relevant information for estimating the translation quality at a character, token, semantic, fluency and adequacy levels before producing the translation. the results were comparable to other qe techniques using nmt hypothesis or probabilities. moreover, we analyzed how the different indicators can heuristically help prioritize segments for pe. on the one hand, the unknown words count is an insightful indicator to select the very first segments to prioritize by choosing segments with one or more unknown words. on the other hand, the maximum semantic similarity is advantageous when the pe task can be applied to more than 40% of the segments. as a result, the combination of both indicators to select segments for pe led to the highest bleu gains above all the qe indicators in most data selection settings. our work opens the door to new perspectives in qe. firstly, we know that the source qe features presented are just a small sample of many other indicators that could be computed to compare a source segment to the nmt training data. nevertheless, this paper highlights the importance of looking back to the training data to evaluate how easily and accurately a segment can be translated by a nmt system. consequently, as it happened with glass-box features in the last wmt qe task, we think that future research on qe supervised models should incorporate these features or any other information that compares the data to be translated against the engine training data.