x T ower: A Multilingual LLM for Explaining and Correcting Translation Errors (2024)

Abstract: While machine translation (MT) systems are achieving increasingly strong performance on benchmarks, they often produce translations with errors and anomalies. Understanding these errors can potentially help improve the translation quality and user experience. This paper introduces xTower, an open large language model (LLM) built on top of TowerBase designed to provide free-text explanations for translation errors in order to guide the generation of a corrected translation. The quality of the generated explanations by xTower are assessed via both intrinsic and extrinsic evaluation. We ask expert translators to evaluate the quality of the explanations across two dimensions: relatedness towards the error span being explained and helpfulness in error understanding and improving translation quality. Extrinsically, we test xTower across various experimental setups in generating translation corrections, demonstrating significant improvements in translation quality. Our findings highlight xTower’s potential towards not only producing plausible and helpful explanations of automatic translations, but also leveraging them to suggest corrected translations.

## introduction neural machine translation (mt) systems have made significant strides in recent years. however, despite their high performance on standard benchmarks, these systems often produce translations that contain errors and anomalies. common methods for evaluating mt quality, such as bleu @xcite , and neural metrics like comet @xcite and bleurt @xcite ), provide only a numerical score reflecting overall translation quality. recent metrics like xcomet @xcite and automqm @xcite highlight error spans to justify their scores but do not offer explanations about the nature of these errors. in-structscore, a recent work by @xcite , leverages large language models (llms) to provide a quality score conditioned on built-in error detection and explanations. however, instructscore primarily functions as a reference-based metric, using explanations as a means to improve score estimates via meta-feedback/finetuning. in this paper, we introduce xtower (figure 1 ), a llm specifically tailored to produce high-quality explanations for translation errors and to utilize these explanations to suggest corrections through chain-of-thought prompting @xcite . xtower is built on towerbase 13b @xcite , a strong open multilingual llm for mt-related tasks. unlike instructscore, xtower can operate without the need for reference translations while also considering information contained in the source sentence. moreover, xtower is designed to be agnostic about the source of error spans, as they can be obtained manually via human annotation or via automatic tools. in this work, we experiment with both. for the automatic case, we leverage xcomet @xcite . this modular approach offers flexibility to experiment with span-level error annotations from various sources, and easily incorporate future improvements in span error detection tools without requiring retraining. we evaluate xtower's explanations both intrinsically and extrinsically. intrinsically, we employ human evaluation to score explanations on two dimensions: relatedness to the error spans being explained ( §4.2) and helpfulness in guiding towards a better translation ( §4.3). extrinsically, we assess xtower's ability to suggest translation corrections ( §5), experimenting with different error span sources @xcite . we compare xtower's performance against leading closed and open llms, such as gpt-3.5 turbo, mixtral 8x7b, and towerinstruct 13b. our findings demonthe source mentions "peru", which is a country in south america, while the translation misinterprets it as "turkey", which can mean a bird or another country. the term "suco de manga" from the source translates to "mango juice". the word "manga" in portuguese can mean "sleeve" or "mango" (fruit). the translation used "head pain" to translate "dor de cabeça" from the source. the more conventional english term is "headache". figure 1 : illustration of our approach. in this example, the input consisting of a source and a translation is passed to xcomet, which annotates the translation with error spans and produces a (discretized) quality score. the full input, marked translation, and quality score are passed to xtower, which, in turn, produces an explanation for each error span along with a final suggestion for a new, corrected translation. ## background in this section, we provide an overview of the key components and concepts relevant to our work. tower. alves et al. (2024) developed a suite of state-of-the-art multilingual llms via continued pretraining of llama2 (touvron et al., 2023) -towerbase-and supervised finetuning for translation-related tasks -towerin-struct. tower is trained to handle diverse tasks such as mt, automatic post-editing, and grammatical error correction. however, it lacks support for error-annotated inputs and cannot produce high-quality, span-level explanations for translation errors. xtower addresses these limitations by extending tower-through distillation and finetuning-enabling it to provide explanations for translation errors and generate corrected translations. ## xtower in this section, we provide details on the methodology behind xtower (figure 1 ), a model built on top of towerbase via distilled supervised finetuning @xcite . ## explaining translation errors in this section, we provide a detailed human evaluation of the quality of the explanations produced by xtower, which are obtained in a more realistic setting via referenceless prompting. ## correcting translations having established that xtower can provide helpful and useful explanations, we turn to prompting it towards obtaining translation corrections. ## related work here, we discuss key related works in the domains of free-text explanations, automatic post-editing, span-level error detection, and the use of llms for translation and error explanation. free-text explanations. recent work has explored methods for generating free-text explanations either by utilizing human-written examples @xcite or by prompting llms @xcite @xcite . however, these explanations are typically produced to understand a model's decision rather than being constrained to justify marked spans in the input. in a similar vein, @xcite propose leveraging dense saliency maps to improve the verbalization of explanations by llms. in contrast, xtower focuses on producing explanations that are tied to specific error spans ( §4.2) and helpful to humans ( §4.3), within the context of mt. ## conclusions in this paper, we introduced xtower, a multilingual llm designed to provide free-text explanations for translation errors and generate corrected translations. by leveraging the strengths of tower and integrating specialized error detection from xcomet, xtower can improve the interpretability of machine translation outputs in an automatic process. our evaluations demonstrate that xtower not only produces high-quality and helpful explanations, as assessed by human evaluation, but can also significantly improves translation quality, especially when combined with accurate error spans. furthermore, we propose a hybrid approach that dynamically selects between using the original translation or querying xtower for a correction, resulting in overall improvements in translation quality for all language pairs. limitations while xtower significantly advances machine translation interpretability, it has various limitations. even though the model's dependence on external error span detector tools like xcomet brings modularity and flexibility, it also introduces pipeline complexity. our evaluation, focused on the few language pairs which have mqm annotations available, may not generalize across all languages and domains. additionally, the computational resources required for distillation and finetuning are substantial, limiting reproducibility for some users. the generated explanations, though helpful, may not always faithfully represent the model's reasoning or effectively guide users. lastly, potential biases in the training data could affect translation and explanation quality, requiring further work to ensure fairness and reliability.