Quality Fit for Purpose: Building Business Critical Errors Test Suites (2023)

Abstract: This paper illustrates a new methodology based on Test Suites (Avramidis et al., 2018) with focus on Business Critical Errors (BCEs) (Stewart et al., 2022) to evaluate the output of Machine Translation (MT) and Quality Estimation (QE) systems. We demonstrate the value of relying on semi-automatic evaluation done through scalable BCE-focused Test Suites to monitor both MT and QE systems’ performance for 8 language pairs (LPs) and a total of 4 error categories. This approach allows us to not only track the impact of new features and implementations in a real business environment, but also to identify strengths and weaknesses in models regarding different error types, and subsequently know what to improve henceforth.

## introduction unbabel's language operations platform blends advanced artificial intelligence with humans in the loop for fast, efficient and high-quality translations that get smarter over time. the company combines machine translation with human post-edition performed by experienced post-editors to translate a variety of content, ranging from customer support to marketing. mt and quality evaluation are at the core of unbabel's business, as the main focus is to provide high-quality translations regardless of the use case or content type. both mt and qe systems have been continuously improving and overcoming existing limita-tions throughout the years. as a result, the need to evaluate their outputs' accuracy and overall performance in error detection grows along with this development process, especially in a business environment where the need to deliver high quality translations without critical errors is paramount. the evaluation of mt outputs can be generally done by following either manual quality assessment procedures with error annotations (such as the multidimensional quality metrics (mqm) framework @xcite ), or automatically by relying on metrics such as bleu @xcite , meteor @xcite , and comet @xcite . in the same way that mt evaluation and complementary metrics are crucial to achieve outputs with better quality, so is the need to evaluate the precision and accuracy of qe systems. to this end, qe systems are oftentimes evaluated against gold annotated data by the pearson correlation score (e.g., @xcite ) and by computing the matthews correlation coefficient (mcc, @xcite ). the main focus of this paper is to overcome the shortcomings of both manual and automatic mt and qe evaluation methodologies in a 'real-life' business environment. we are able to achieve this through a semi-automatic approach that relies on mt test suites @xcite in a production setting. the mt test suites proposed here follow the concept of bces @xcite and consist of proprietary corpora with mqmannotated data @xcite . with this in mind, we demonstrate how mt test suites can be leveraged to provide a semi-automatic method of mt evaluation and how they can be a good compromise between manual and automated metrics by taking into account errors that are harmful in a business environment. in this paper, we seek to address the following: for this purpose, we present large-scale and fine-grained mt test suites for 8 lps with english as source language for all possible combinations. as we base our approach on the concept of bces @xcite , the mt test suites proposed here will be called bce test suites. ## related work the mt field witnessed a breakthrough in the quality of translations with the rise of neural mt (nmt). as such, the need for evaluating the performance of different systems increased concurrently. there are two major types of approaches when evaluating mt systems: manual and automated metrics. regarding manual metrics, one distinctive method has been widely adopted in an attempt to standardize the evaluation process: the mqm framework @xcite , which provides a hierarchical categorization of issue types and dependencies regarding errors in translation outputs. each error is annotated by human annotators with a precise issue type, along with the level of severity that affects the target text and its perceived quality. there are three severity levels an error can be classified as: minor, major, and critical. however, it is important to stress the difference between critical errors from a linguistic quality perspective and errors related to the perceived quality of the translation. while critical linguistic errors severely impact the grammaticality of the text, errors that disturb the perceived quality of a translation are considered as business critical errors (bces) @xcite . this is due to the fact that they not only include errors that are considered linguistically critical, but also errors that may cause additional damage in a customer-focused environment. in addition to manual processes of mt evaluation, automatic metrics have also been commonly adopted in the industry to assess the mt outputs' quality along with the mt systems overall performance. two examples of these metrics, among the most commonly used ones, are: bleu @xcite , that estimates a translation's quality value by solely relying on precision, and comet @xcite ) -a widely-used recent metric developed by unbabel. comet is a neural framework that allows multilingual mt evaluation and that highly correlates with human judgement. another way to measure systems' performance is by automatically estimating the quality of the translation without access to a reference. specia et al. (2009) and kepler et al. (2019 ) are able to achieve this through the use of qe metrics. while specia et al. (2009) estimate quality by relying on a continuous score, kepler et al. (2019) took a step forward for qe tools and created a new open source framework named openkiwi foot_0 . openkiwi, created by unbabel, achieved stateof-the-art results in word-level qe at the time. following that, unbabel also won the wmt 2022 shared task on quality estimation @xcite with an extension of the comet @xcite ) framework called cometkiwi @xcite , which merges the benefits of comet's multilingual training features with openkiwi's predictor-estimator architecture. despite all the advancements of the automatic evaluation approaches, the existing solutions fail, to some extent, to detect bces. in order to relax this issue, there are several approaches to data augmentation, such as augly @xcite and, more recently, alves et al. (2022) who proposed a new sentence-level multilingual augmentation (smaug) framework that generates critical errors in translations in order to improve robustness of state-of-the-art mt metrics. although both evaluation methods allow for a performance comparison of mt systems, each one shows different advantages and constraints. while automated metrics are unable to provide information about translation error types, they provide a reproducible generic score of correctness @xcite in a time-and cost-efficient manner. on the other hand, manual evaluation is time-consuming and less scalable than automatic methods as it consists of plain human judgement. nonetheless, manual evaluation is able to provide evaluations that are much more fine-grained and sensitive to nuanced errors. with this in mind and in an attempt to achieve a more detailed qualitative analysis on performance evaluation, a semi-automatic approach that relies on previously revised test sentences to evaluate performance of mt systems was developed in order to merge the advantages of both methods. these test sentences are specifically assembled to obtain corpora of controlled examples, i.e., to obtain test suites. the chosen examples in test suites are referred to as the gold-standard data and are used for diagnostic evaluation of mt systems. depending on the type of evaluation desired, test suites can be adapted to fit different purposes. as such, they can focus on more specific linguistic phenomena @xcite or on generic system's evaluation, as well as being created upon fabricated examples or representative content translated by the mt systems. thus, their construction is required to follow a linguistically motivated approach, which allows them to be used for comparative analysis between systems @xcite . in sum, by combining manual evaluation with automated metrics, it is possible to obtain values that are much more precise and accurate at describing systems' performance and at identifying the most problematic structures. ## methodology as a means of measuring translation quality, unbabel performs mqm annotations by using a proprietary mqm-compliant typology adapted from the original mqm proposed by @xcite . annotations are performed by unbabel's professional community of annotators, composed of professional translators and linguists with significant experience in linguistic annotations and the detection of translation errors. the result of this process is not only an mqm score that indicates the quality of a given translation, but also annotated data with precise information about the types of errors and the associated severities that occur in mt outputs. besides this, unbabel developed the concept of bce @xcite , a subset of error categories that can have direct business implications for customers and that would otherwise render a translation 'unfit', regardless of perceived linguistic quality. with mqm annotations we are able to identify the relevant bces produced by mt systems and we use them as the basis to build bce test suites. the bce test suites proposed here consist of a total of 8 lps ( with this, we produced a total of 11,481 test sentences, in which each test sentence represents one single error type. for each one of the 4 categories that compose the bce test suites, we aimed at a minimum of 50 test sentences. ## experimental setup the main purpose of the bce test suites is to evaluate the ability of mt and qe systems to avoid or detect certain types of errors that can potentially be harmful to customers, according to the type of content and the level of quality expectations related to it. in this paper, we aim to test and measure the behavior of such systems in a real business scenario, especially in regards to the implementation of new features in customers' mt systems and a new mqm-qe model. ## conclusions and future work in this work, we present a methodology to build test suites that are tailored to address business critical errors @xcite and how they could potentially harm customers in a business setting. we demonstrated that it is possible to use a dataset of translation errors annotated by following the mqm framework @xcite of 'reallife' machine translation errors to build comprehensive test suites for several lps in order to evaluate the performance of both mt and qe systems. as shown in section 5.1, relying on the bce test suites scores alongside the automatic metrics to decide whether a model should be deployed to production or not brings great value to the robustness of the model, hence results about bce test suites accuracy will be added to the automatic deployment criteria. bce test suites are also a valuable part of the qe evaluation pipeline, highlighting errors that are important in a business setting. for future work, we would like to extend the information in the test suites to include all errors in the sentence, so we can measure precision-based metrics as well.