Towards General Error Diagnosis via Behavioral Testing in Machine Translation (2023)

Abstract: Behavioral testing offers a crucial means of diagnosing linguistic errors and assessing capabilities of NLP models. However, applying behavioral testing to machine translation (MT) systems is challenging as it generally requires human efforts to craft references for evaluating the translation quality of such systems on newly generated test cases. Existing works in behavioral testing of MT systems circumvent this by evaluating translation quality without references, but this restricts diagnosis to specific types of errors, such as incorrect translation of single numeric or currency words. In order to diagnose general errors, this paper proposes a new Bilingual Translation Pair Generation based Behavior Testing (BTPGBT) framework for conducting behavioral testing of MT systems. The core idea of BTPGBT is to employ a novel bilingual translation pair generation (BTPG) approach that automates the construction of high-quality test cases and their pseudoreferences. Experimental results on various MT systems demonstrate that BTPGBT could provide comprehensive and accurate behavioral testing results for general error diagnosis, which further leads to several insightful findings. Our code and data are available at https: //github.com/wujunjie1998/BTPGBT.

## introduction in recent years, machine translation (mt) systems have achieved significant advancements in translation performance. yet current mt systems are still brittle and could generate erroneous translations that lead to severe commercial losses 1 , which makes error diagnosis of mt systems a crucial task to be solved. behavioral testing, which has been widely applied in software engineering research to test complex systems, is a desired direction to tackle this issue. generally, behavioral 1 https://www.androidpolice.com/ google-translation-mistake-io/ in order to tell new stories to the market, zuckerberg told the story about "the metaverse". 为了给市场讲出新的故事，扎克伯格讲了"元宇宙"。 为了向市场讲述新的故事，扎克伯格讲述了关于"metaverse"的故事。 𝒙′: 𝒓′: 𝒚′: capability: ner in order to tell new stories to the market, zuckerberg told the story about "meta-universe ". 为了给市场讲出新的故事，扎克伯格讲了"元宇宙"。 为了向市场讲述新的故事，扎克伯格讲述了关于"元宇宙"的故事。 𝒙: 𝒓: 𝒚: in order to tell new stories to the market, zuckerberg told the story about "<@xmath0. 为了给市场讲出新的故事，扎克伯格讲了"<@xmath1: 2 𝒓: btpg generates (𝒙 ! , 𝒓 ! ) figure 1 : an example of behavioral testing the deepl translator on the capability of translating named entities (ner). x is the original input and r is its reference. the masked counterparts of x and r are denoted as x and r respectively, with the to-be-masked segments highlighted in boldface. a generated test case, x ′ , replaces the named entity "meta-universe" in x with "the metaverse", and its crafted pseudo-reference is denoted as r ′ . deepl's outputs for x and x ′ are denoted as y and y ′ . deepl does not pass the behavioral testing on x ′ since it outputs an erroneous y ′ that directly copies "metaverse" instead of translating it to "元宇宙", which is identified by the substantial difference in the comet scores between y and y ′ . testing probes various system capabilities for error diagnosis by inspecting input-output behaviors on test cases with targeted edits, irrespective of knowledge about the system's internal structure @xcite . therefore, it is suitable for providing finegrained evaluations of different capabilities and diagnosing errors for mt systems. there are systematic studies of behavioral testing on classification tasks such as sentiment analysis @xcite , hate speech detection @xcite and clinical outcome prediction @xcite . however, implementing behavioral testing in mt tasks poses significant challenges. the main reason is that it is usually difficult to automatically obtain the ref-erence translation for a test case modified from a source sentence, making it difficult to automatically judge the quality of the translation of the test case. to avoid this challenge, existing work attempts to judge the translation quality of such test case without its reference translation, yet they can only diagnose translation errors related to specific types of edits that target certain capabilities. for example, @xcite ; @xcite ; @xcite ; @xcite only diagnose translation errors on test cases with the editing of a single noun or adjective word, and @xcite ; @xcite ; @xcite can only diagnose incorrect translation of noun phrases, quantities or currency units that is related to the edits on them. to address these challenges, this paper presents bilingual translation pair generation based behavior testing (btpgbt), a novel framework for behavioral testing in mt systems. the core idea of our framework is the novel bilingual translation pair generation (btpg) approach that can automatically generate high-quality test cases and their pseudo-references from standard mt test sets, 2 which allows the diagnosis of general translation errors in test cases targeting various mt system capabilities. as shown in figure 1 , btpg takes a source sentence and its reference as an input translation pair, then masks specific aligned segments ("meta-universe" and "元宇宙" in figure 1 ) in both sentences targeting the capability that needs to be tested (ner in figure 1 ). only masking aligned segments enables btpg to make the best use of the structure information of unmasked positions in the original source and reference during generation, which enhances its generation quality. btpg then generates a new bilingual translation pair as the test case and its pseudo-reference by using chat-gpt 3 , a large language model proposed by ope-nai that has shown impressive performances on various nlp tasks, to infill the masked segments in both sentences at once. in this way, the test case and its pseudo-reference could have similar quality to human constructed ones, as illustrated in §4.2. with the high-quality test cases and their pseudoreferences crafted by btpg, we can conduct behav-2 our framework indeed involves a test set with references, but this is not a very strong requirement since there are many off-the-shelf test datasets and in particular this is a standard setting in behavioral testing. for instance, the pioneered research about behavioral testing @xcite and many follow-up works @xcite all use a test dataset with ground-truth labels. ## btpgbt framework to solve issues in previous works, we propose a novel btpgbt framework to conduct behavioral testing for general translation error diagnosis. given a source sentence x and its reference r, the core idea of btpgbt is to automatically edit one or more segments in x and r to construct various test cases x ′ and their pseudo-references r ′ targeting specific capabilities of mt systems. then we can diagnose these systems following the steps described in §2. in the following sections, we first describe how we implement the above core idea, then illustrate the capabilities btpgbt tests as well as how to generate test cases targeting these capabilities for behavioral testing. ## experiments we conduct experiments on the english-chinese translation task. first, we implement both auto-matic and human evaluations to measure the quality of the btpg method. next, we apply btpgbt to diagnose six mt systems to show the effectiveness of our proposed behavioral testing framework, and perform in-depth analysis on the evaluation results to obtain some insightful findings. ## conclusion in this paper, we introduce a novel behavioral testing framework btpgbt to diagnose general translation errors for mt systems. the key idea of btpgbt is to auto-generate test cases and their pseudo-references, which facilitates the diagnosis of translation errors targeting various capabilities of mt systems. to this end, we design the btpg approach to craft test cases and pseudo-references, whose effectiveness has been proven by extensive experiments. experimental results on six different mt systems further demonstrate that btpgbt can provide general and faithful behavioral testing results and lead to some insightful findings.