Targeted Source Text Editing for Machine Translation: Exploiting Quality Estimators and Large Language Models (2025)

Abstract: To improve the translation quality of “black-box” machine translation (MT) systems,we focus on the automatic editing of source texts to be translated.In addition to the use of a large language model (LLM) to implement robust and accurate editing,we investigate the usefulness of targeted editing, i.e., instructing the LLM with a text span to be edited.Our method determines such source text spans using a span-level quality estimator, which identifies actual translation errors caused by the MT system of interest, and a word aligner, which identifies alignments between the tokens in the source text and translation hypothesis.Our empirical experiments with eight MT systems and ten test datasets for four translation directionsconfirmed the efficacy of our method in improving translation quality.Through analyses, we identified several characteristics of our method andthat the segment-level quality estimator is a vital component of our method.

## introduction in the last decade, the quality of machine translation (mt) outputs has been significantly improved as a result of the advancements of neural mt (nmt) and large language models (llms) and the accumulation of parallel data in the community. a number of new techniques for further improving translation quality, i.e., reducing translation errors, have been presented at conferences; however, proprietary mt services have tended to remain state of the art, presumably thanks to undisclosed technologies and massive in-house data. such "black-box" systems are, in general, difficult to adapt for users' niche use cases in which texts with specific content domains or text styles are to be translated. to obtain better translations using black-box mt systems, several strategies have been proposed. one such strategy is "pre-editing," i.e., editing given source texts to improve their translation by an mt system of interest. although studies on automatic pre-editing have long been conducted ( §2), two issues remain. first, existing methods have only limited editing ability. various types of source text editing can potentially improve its translatability @xcite . however, in past studies, researchers have addressed only specific linguistic complexities or performed the regeneration of entire texts indiscriminately. another issue is that researchers have performed pre-editing without referring to the actual translation generated by the mt system, despite the proven effectiveness of editing source text with reference to actual translation errors @xcite . different mt systems have different error tendencies; thus, editing expressions that the mt system can translate well would result in new translation errors. in this study, we automate "targeted source text editing" using (a) quality estimation models to determine what to edit on the basis of the translation errors caused by a target mt system and to search for a better translation and (b) llms to realize various types of editing as in @xcite . in our method ( §3), a given source text is translated by the mt system, and errors in the output are identified by a span-level quality estimator. then, using an llm, our method edits the source text with a text span annotated as the source of the severest translation error. guiding the editing process with a trigger error does not guarantee that the mt system can translate the edited text better @xcite . thus, our method searches for a better translation by repeating text editing and mt, relying on a segment-level quality estimator. our empirical experiments with eight mt systems and ten test datasets for four translation directions confirmed the efficacy of our method in improving translation quality ( §5). our analyses also revealed several characteristics of our method, including the diverse impact depending on the mt system and dataset, the necessity of improving the segment-level quality estimator, and controlled editing realized by tailored instruction ( §6). ## previous work mt systems and services have gradually pervaded our lives, and have been incorporated into the human-centered translation production process adopted by translation/language service providers (iso/tc37, 2017). before they became sufficiently practical, researchers examined several approaches to human-mt interaction. @xcite proposed the editing of source texts motivated by translation errors; the method was later named "targeted paraphrasing" by @xcite . inspired by previous studies on targeted paraphrasing, @xcite conducted manual investigations into the pre-editing strategy for exploiting black-box services based on statistical and neural mt. through incrementally performing source text editing in four content domains referring to mt outputs, they found that most (80%-100%) of the source texts could eventually be edited so that they will lead to no translation errors and that human editors have performed diverse types of edits, not only limited to paraphrasing, that can improve translation quality. automatic "pre-editing" methods, which have been studied for three decades, can be classified into two groups. the first group focuses on specific linguistic phenomena that are difficult to translate, such as low frequency words, subject ellipsis, and long sentences, and avoids them relying on a set of rewriting rules based on morpho-syntactic information, corpus statistics, and neural language models @xcite @xcite @xcite @xcite @xcite . however, each of these methods only covers a specific type of editing, among the diverse promising ones. early methods are difficult to replicate for other source languages because of their heavy reliance on hand-crafted rules and resources. another line of research has attempted to regenerate entire source texts by regarding the task as monolingual translation and applying datadriven sequence-to-sequence decoding methods @xcite @xcite . the performance of this approach is dominated by the characteristics and quantity of training parallel data. since no parallel data have been specifically tailored for the purpose of pre-editing, except for small collections for manual analyses and evaluation, researchers have used monolingual parallel data that exhibit other monolingual tasks, such as text simplification and text revision, or synthetic parallel data generated by back-translating bilingual parallel data and round-trip translation of monolingual data. this approach has been proven effective for rule-based and statistical mt systems @xcite @xcite ; in contrast, it does not necessarily work for nmt systems @xcite . recently, @xcite examined the utility of llms for source text editing. they compared several editing strategies and identified that the instruction for text simplification and selection based on quality estimation are effective. unlike manual investigations @xcite , both of the aforementioned groups of methods do not refer to the translation errors caused by the mt system of interest. although a linguistically motivated pre-edit should be helpful in general, excessive editing of translatable expressions would introduce new translation errors. ## targeted source text editing unlike existing "pre-editing" methods, we propose the editing of given source texts to avoid actual translation errors that the target mt system makes, i.e., the automation of the manual investigation process @xcite . algorithm 1 shows the overall procedure of our method. given a source text src 0 , our method first translates it using an mt system of interest t , and evaluates the quality of the generated hypothesis hyp 0 with reference to src 0 using a quality estimator q. the pair of hyp 0 and its score initializes the best result (steps 1-3). the open list of errors to be edited is also initialized with translation errors in hyo 0 identified by an error detector e (steps 4-5). then, for pre-defined iterations n or until no errors remain (step 7), our method repeats the following steps: (a) identify the severest error and corresponding source text span (steps 8-9, §3.1), (b) edit the identified source span (step 10, §3.2), (c) translate the edited source text and evaluate the new hypothesis (steps 11-12, §3.3), and (d) search for the best translation (steps 8, 13-17, §3.4). algorithm 1: proposed error-informed source text editing method. input :original source text src0, translator t , quality estimator q, error detector e, maximum iteration n , aligner a, paraphraser p output :best translation best_hyp 1 hyp 0 = t.translate(src0) 2 @xmath0.evaluate(src0, hyp 0 ) 3 @xmath1.detect_errors(src0, hyp0) 5 @xmath2, hyp 0 , errs⟩] // open list 6 @xmath3, hyp, err ⟩, @xmath4.propagate_error (src, hyp, err ) 10 @xmath5.paraphrase(src ann ) 11 hyp @xmath6.translate(srci) 12 @xmath7.evaluate(src0, hyp i ) 13 if @xmath8.detect_errors(srci, hyp i ) 17 cands.append (⟨srci, hyp i , errs⟩) 18 @xmath9 ## preliminary experiments we determined the detailed settings using a small set of japanese-to-english translation examples and one mt system. henceforth, resources used in our experiments, including datasets, pre-trained model checkpoints, and tools, will be presented in this manner. see appendix a for their details. for this purpose, we used an nmt model pretrained on jparacrawl @xcite (the big model) and four sets of japanese-english parallel data: (a) the development data of aspec @xcite (abstracts of scientific papers), (b) the test data of wmt22 @xcite (mixture of several domains), (c) the test data of mtnt @xcite ) (users' posts on social media), and (d) the development data of the kyoto free translation task (benchmark splits of the japanese-english bilingual corpus of wikipedia's kyoto articles; henceforth, kftt). first, we translated the japanese side of these datasets into english using the mt model and fairseq @xcite , and detected translation errors using a span-level quality estimator, xcomet-xl @xcite . we then randomly extracted 25 text pairs from each dataset that contained at least one "critical" or "major" error. referring only to the sampled 100 (= 25 × 4) text pairs, we determined the details of our method, including the combination of the span-level quality estimator and word aligner (appendix c.1), the llm for source text editing (appendix c.2), and the prompt template (appendix b.1). to identify text spans in the source texts to be edited, our method propagates errors identified by xcomet-xl, relying on optimal transport implemented by otalign @xcite . we determined the alignments between the tokens in the source text and hypothesis using contextual token embeddings obtained by infoxlm-base @xcite , uniform distribution as the mass for each token, cosine distance between token embeddings as the cost function, sinkhorn algorithm, and 0.1 as the weight for the entropy-based regularizer. for source text editing, we used llama-3.1-swallow-70b-instruct-v0.1 (henceforth, llama-swallow) @xcite , an llm trained on massive text data of the source language, i.e., japanese, and devised a prompt template for it, including the way of specifying the source text span to be edited and providing a one-shot paraphrase example. such an example was randomly sampled from the 71 paraphrase examples in japanese taken from a taxonomy of paraphrases. 1 the instruction was formatted by applying a chat template using the language model evaluation harness. to select the best translation hypothesis among those derived from different versions of source 1 https://paraphrasing.org/paraphrase.html texts, we also used xcomet-xl, a reference-free quality estimation metric. through the preliminary experiments, we observed that the translation quality achieved by our method saturated up to five iterations (appendix c). this is fewer than the 5.4-21.8 iterations required to obtain an acceptable translation in a manual investigation @xcite because of the limited ability of our method compared with humans. the advancement of each component, as well as their tight integration, should improve the ability of our method. ## evaluation to confirm the efficacy of our method on translation quality, we conducted experiments. although most components of our method are multilingual, we had only japanese and english speakers to write prompt templates for source text editing and translation with llms. therefore, we evaluated the applicability of our method on japanese-to-english (@xmath10, japanese-to-chinese (@xmath11, englishto-japanese (@xmath12, and english-to-chinese (@xmath13. ## analyses to better understand the characteristics of our method, we conducted several analyses, focusing on the erroneous test subsets ( §5.1.3). ## conclusion as an approach to exploiting black-box mt systems, we focused on automatic and targeted source text editing. to overcome the two issues that remain in the literature, i.e., the limited ability of editing and unawareness of actual translation errors, we used llms, expecting that they would have sufficiently high competence to realize diverse types of edits that can improve translation quality @xcite and a segmentlevel quality estimator as in a concurrent work @xcite , and implemented targeted paraphrasing @xcite by harnessing a span-level quality estimator (error detector) and a word aligner. our experiments with eight mt systems and ten test datasets for four translation directions confirmed the efficacy of our method in improving translation quality, while the non-targeted counterpart (llm-nt) also achieved a rivaling performance. through the analyses, we identified that the impact of our method varied depending on the mt system and dataset, and that the segment-level quality estimator is the vital component that requires further improvements. future work includes improving each component of our method, while simplifying and speeding up the whole pipeline. since our method focuses on source texts that lead to translation errors according to an error detector, applying llm-nt to other error-free segments will be a straightforward way for improving translation for entire datasets. only prompt templates are language dependent; hence, we plan to evaluate the applicability of our method to other translation tasks as well as other mt systems, including proprietary ones. we are also interested in assessing the applicability of the proposed method to other text-to-text tasks, including text summarization and text simplification.