Automating Behavioral Testing in Machine Translation (2023)

Abstract: Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine Translation (MT) is currently restricted to largely handcrafted tests covering a limited range of capabilities and languages. To address this limitation, we propose to use Large Language Models (LLMs) to generate a diverse set of source sentences tailored to test the behavior of MT models in a range of situations. We can then verify whether the MT model exhibits the expected behavior through matching candidate sets that are also generated using LLMs. Our approach aims to make behavioral testing of MT systems practical while requiring only minimal human effort. In our experiments, we apply our proposed evaluation framework to assess multiple available MT systems, revealing that while in general pass-rates follow the trends observable from traditional accuracy-based metrics, our method was able to uncover several important differences and potential bugs that go unnoticed when relying only on accuracy.

## introduction automatic evaluation metrics such as bleu @xcite and comet @xcite are the primary means of measuring the translation quality of mt systems. researchers and practitioners rely on them for comparing systems, detecting regressions, and making deployment decisions. this poses an important concern: such metrics typically aggregate the performance of systems across a set of sentences into single scores. unfortunately, these metrics by design tend to overlook specific infrequent but important error cases, making it difficult to reliably detect such issues in practice. behavioral testing, originally developed as a type of software testing @xcite , has been proposed as an approach that can alleviate such kinds of problems in natural language processing @xcite . behavioral tests focus on assessing a system's fine-grained linguistic capabilities by validating input-output behavior in a controlled fashion. however, there are currently two major limitations that arise when attempting to apply behavioral testing to mt. first of all, behavioral testing was originally designed for evaluating systems characterized by a relatively small output space. for instance, @xcite investigate sentiment classification, duplicate question detection, and machine comprehension. in contrast, the output space of mt systems grows exponentially as tokens are generated. secondly, behavioral testing often requires rigid templates to create examples and their ## source sentence generation to create source sentences for testing a certain language property, we pose several desiderata: sentences should be diverse (e.g. not rely only on a handful of templates), natural, numerous enough to yield statistical significance, and contain a property value associated with our tested property. note that existing approaches often struggle to generate diverse test sets due to the reliance on hand-crafted templates @xcite . to overcome this shortcoming, we design a general template for prompting llms, in our case chatgpt 3 , openai's model built on instructgpt @xcite . this allows us to generate diverse source language sentences that contain property values suitable for testing different capabilities (see prompt 4 in figure 2 ). we instantiate the prompt once for every language property that 3 gpt-3.5-turbo api accessed on may 2023. 4 we set @xmath0.9, @xmath1. candidates examples @xmath2, km @xmath3, w meters per @xmath4, m/s to simplify the later verification step, we generate sentences that contain exactly one such property value x v . 5 we generate source sentences with brackets around the property value for easy parsing. a possible test sentence for the property of translating decimal numbers might look as follows: the company received [ property value 4200.4 ]c. note that brackets are removed before passing the sentence to the mt model. we apply basic filters to remove duplicated sentences, examples with more than one property value, or those composed of more than one sentence. we repeatedly feed the same prompt to the llm, and stop the generation process when reaching 1,000 sentences after filtering. our experiments ( §9) indicate that chatgpt is able to generate sentences of adequate quality and diversity. ## candidates generation next, in order to be able to verify whether an mt system correctly translated the property value in the source sentence, we automatically generate valid translation candidate sets for each property value. for some properties, such as number translation, we create exhaustive or near-exhaustive candidate sets. for other properties where the number of valid translations would be too big to do so, we instead create contrastive candidate pairs that demonstrate desired and undesired behavior. note that candidate sets only need to be created once and can then be re-used for every tested system. ## evaluation metrics having established pass-fail detection for individual sentences, the final step is to compute aggregated pass rates across test sets. appealingly, pass rates are naturally expressed as percentages, making them intuitive to interpret. ## properties to test we design a number of tests and use our proposed framework to evaluate mt models in multiple properties. the chosen properties, also studied in the literature @xcite , have two important qualities that make them useful for evaluating translation systems: vital for producing high-quality translations, yet posing a challenge when assessing through conventional evaluation metrics. numbers. we conduct independent assessments for integers (e.g. 1887), decimals (e.g. 154.32), and large numbers (e.g. 200 billion). large numbers have the format "integer/decimal million/billion/trillion". we create near-exhaustive candidate sets of valid number translations and check if the translation matches any candidate. physical units. we build near-exhaustive candidate sets for evaluating the translations of diverse units including those related to weight, length, time, or temperature inter alia (e.g. inches). translations are evaluated by string matching. emojis, names, and web terms. via string matching we check whether the translated text retains the same property instantiation found in the source text. candidate sets for these tests are thus considered to be exhaustive. currencies. we consider currencies appearing in the iso code format (e.g. eur). near exhaustive candidate sets are built allowing translations into the same iso code, variations of the currency name or its symbol (e.g. for @xmath6: eur/euro/euros/c), then a string matching passfail detection is employed. idioms. idiomatic expressions pose significant challenges for mt systems due to their non-literal nature and potential large sequence length. we use idioms as a test bench for the use of contrastive candidate pairs (incorrect literal translation candidate vs. correct meaning translation) and semantic similarity detection procedure. ## models comparison in this section, we introduce the tested models and present results obtained via standard metrics as well as our proposed framework. ## reliability of the proposed approach to assess the reliability of the proposed approach, in this section we analyze the robustness of source sentence generation and pass-fail detection. ## related work recent works have applied behavioral testing for evaluating machine translation systems. @xcite designed tests for numerical translation capabilities by relying on fixed templates for source sentence generation. @xcite proposed salted, a set of manually designed error detectors that are applied to millions of sentences from standard datasets. beyond behavioral testing, a large number of challenge sets have been developed for machine translation @xcite . although useful, most of these evaluation tools require major human efforts for creation, evaluation, or expanding to other languages. although there have been attempts to automatize the creation of behavioral tests @xcite , this has been limited to simple nlp tasks. our work also relates to the use of llms as evaluators for machine translation systems @xcite , as well as for text generation in a broader sense @xcite , which extend the growing body of research on multi-dimensional text generation evaluation @xcite . behavioral testing aims to evaluate the behavior of systems under realistic conditions, contrasting it from the literature on adversarial data generation @xcite . ## conclusions in this work, we have presented a method that automates the creation of behavioral tests to perform fine-grained evaluation of mt systems capabilities. we use large language models to generate source sentences composed of fragments of specific language properties (integers, web terms, etc.), as well as translations of these properties. for property types formed by multiple words, we further extend the proposed method into a contrastive setting and show its usefulness in evaluating idiomatic expressions. to the best of our knowledge, our research represents the first attempt to develop mt behavioral tests by leveraging llms. finally, we apply the proposed framework to evaluate open-source models on three language pairs.