SALTED : A Framework for SA lient Long-tail Translation Error Detection (2022)

Abstract: Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems. Examples include translation of numbers, physical units, dropped content and hallucinations. These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems. Consequently, it is important to have visibility into these problems during model development.Towards this end, we introduce SALTED, a specifications-based framework for behavioral testing of NMT models. At the core of our approach is the use of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output. These detectors provide fine-grained measurements of long-tail errors, providing a trustworthy view of problems that were previously invisible. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model fine-tuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.

## introduction the development of machine translation (mt) systems is typically guided by performance metrics @xcite computed on small, curated test sets, wherein system quality is often reduced to a single number. while metrics are useful for characterizing the average performance of a system, they do not provide finegrained visibility into rarer error categories. as a result, researchers do not have a reliable way to gauge whether and to what extent a system may exhibit a wide range of negative behaviors, such as hallucinations, dropped content, or the sporadic mistranslation of important information such as names or physical units. however, such salient 'long-tail' errors undermine the reliability of mt systems, and are increasingly important to curtail in an era where mt output is often indistinguishable from that of humans @xcite . a second related problem is that many of these behaviors are rare enough that they will not be observed on standard test sets (which typically only number a few thousand sentence-pairs), even if reliable detection via references is feasible. owing to this rarity, the detection methods that require source-reference pairs are not useful, besides being non-scalable. consequently, a real missing feature in machine translation evaluation is having finegrained analysis and measurements that could scale to large, unannotated datasets. in this work, we propose salted as a framework to tackle these challenges. our main contributions are as follows: ## behavior specification for mt if we cannot specify the correct behavior, we cannot verify output correctness @xcite . therefore, the first step of constructing a detector is to specify the desired behavior that the model must satisfy with respect to the translation of certain salient content or property. ## designing detector algorithms building detectors from specifications is an iterative process. a first implementation for most content types inevitably yields large numbers of false positives. it is here that our focus on precision over recall serves as a useful guide. in addition to producing detectors whose results are more trustworthy, this principle simplifies the task, since we can narrow in on a subset of settings that could be detected with a high certainty of correctness. here we illustrate this process with an example. ## evaluations using salted having constructed seven high-precision detectors, we now wish to apply them to commercial systems to investigate whether we can discover any problems. we take a sample of 100k sentences from a larger 1m monolingual corpus (detailed in section 6.1) and translate them with google, microsoft, and amazon's systems by way of their paid public apis. these results demonstrate that long-tailed errors are quite pervasive across nmt systems, despite being very rare (only 0.3% incidence rate for google, based on ## system comparisons & data filtering a general trend in nmt is the susceptibility of trained systems to even small amounts of noisy data @xcite . we investigate whether detectorsoptimized for error precision, rather than recall-can work as effective filters to improve systems. ## metamorphic testing the low incidence rates of long-tail errors necessitate large amounts of data in order to elicit them. the salted framework further addresses this problem through metamorphic testing, wherein new test inputs are produced by modifying an input instance in systematic ways and the outputs are then tested for correctness through detectors. ## conclusions in this paper, we have advocated for and demonstrated the utility of a principled, specificationsbased approach to reliably flag salient long-tailed mt errors through high-precision detectors. we introduced an iterative, precision-driven process for developing such detectors and applied it on seven classes of mt errors, eliciting a range of errors from state-of-the-art research and commercial systems. although the manual development of such detectors incurs significant cost, the resulting payoff is high with the constructed detectors applicable universally across different systems and datasets. further, we demonstrated the utility of salted for four different use cases in mt: for obtaining reliable measurements of salient long-tailed errors in translations of arbitrary monolingual data, for corpus filtering, for system comparisons, and for fixing token-level errors through a synthetically generated meta-corpus that teaches the model to learn correct behaviors. we hope that our work serves as a useful step towards more reliable mt. 5170 while our work highlighted a number of longtailed errors in state-of-the-art translation mt systems, a limitation of the approach is the effort required in manually defining specifications and then constructing detectors based on the defined specifications. our proposed approach might also face hurdles in adoption since, to the best of our knowledge, the idea of specifications without any references is novel in large-scale mt evaluation, and has been mainly developed and used in software/hardware engineering research. another limitation of the work is that fixing salient long-tail errors is only feasible when the errors are mistranslations, rather than dropped content (e.g., dropped clause or phrase). if the salient errors are generated by long chunks of source sentences remaining untranslated by the model, then the proposed finetuning approach will not improve such translations.