Enhanced Zero-Shot Machine Translation via Fixed Prefix Pair Bootstrapping (2025)

Abstract: Zero-shot in-context learning allows large language models (LLMs) to perform tasks using only provided instructions. However, pre-trained LLMs often face calibration issues in zero-shot scenarios, leading to challenges such as hallucinations and off-target translations that compromise output quality, particularly in machine translation (MT). This paper introduces a new method to improve zero-shot MT using fixed prefix pair bootstrapping. By initializing translations with an accurate bilingual prefix pair at the start of both source and target sentences, this approach effectively guides the model to generate precise target-language outputs. Extensive evaluations across four model architectures and multiple translation directions demonstrate significant and consistent improvements, showcasing the potential of this straightforward strategy to enhance zero-shot MT performance.

## introduction large language models (llms), pre-trained on vast unlabeled datasets, exhibit a remarkable ability known as @xcite . this enables them to adapt to tasks using textual demonstrations, eliminating the need for updates to their underlying parameters @xcite . unlike traditional task-specific fine-tuning, prompting involves crafting instructions that guide llms to solve tasks directly. when combined with icl, where a few labeled examples are included in the input as demonstrations, this approach significantly enhances performance. however, the effectiveness of icl is highly sensitive to the choice of these demonstrations @xcite , creating challenges in practical scenarios where user queries are unpredictable and prior knowledge is unavailable. preparing even a small set of examples for "fewshot" prompts can be labor-intensive and imprac-tical for new tasks, especially in real-world applications. as a result, there is growing interest in zero-shot icl, which removes the reliance on preprepared examples and focuses on enabling llms to handle tasks solely based on instructions. this shift toward zero-shot approaches aims to streamline the use of llms in dynamic and resourceconstrained environments. zero-shot icl allows models to perform tasks based solely on instructions, without relying on labeled examples or demonstrations. this capability harnesses the innate strengths of llms to interpret and generate content using natural language instructions and context @xcite . in mt, zero-shot prompting enables llms to translate a source sentence into a target language based solely on provided instructions. however, a common challenge in zero-shot mt is the offtarget problem @xcite , where the model generates translations that stray from the target language. to address this, we propose a simple yet effective method that requires only a single bilingual word or phrase pair from a dictionary, used as a prefix for both source and target sentences. this approach avoids reliance on comprehensive dictionaries, making it especially beneficial for lowresource languages where such resources are scarce. by introducing fixed prefix pairs, we steer llms to initiate translations correctly and maintain consistent generation in the target language. our evaluation on the flores-101 dataset @xcite across four models and four language pairs demonstrates significant improvements in translation performance, highlighting the effectiveness of our method. ## related work zero-shot icl for mt using llms has emerged as a transformative approach, leveraging llms' ability to perform translation tasks without explicit training on parallel corpora. this method holds significant potential for low-resource languages and specialized domains @xcite @xcite . however, challenges such as hallucinations and the off-target problem, where translations deviate into unintended languages, persist. several studies have investigated these challenges. for example, some prior works @xcite @xcite analyzed the off-target problem, proposing techniques to mitigate it @xcite . @xcite explored factors influencing performance variations in zero-shot neural mt (nmt) across diverse language pairs, models, and training configurations. @xcite introduced multilingual vocabularies in decoders to isolate language-specific tokens, reducing the likelihood of incorrect language outputs. other works focused on enhancing zero-shot mt through innovative prompting and decoding strategies. @xcite proposed adding output text distribution signals to improve zero-shot prompting for gpt-3, achieving competitive results with few-shot methods. this study also revealed the asymmetric impact of perturbing source and target sides, emphasizing the importance of targetlanguage continuity in translation quality. similarly, the mtcue framework reinterpreted contextual attributes as text, enabling zero-shot control of extra-textual features like formality, significantly improving translation quality over non-contextual baselines @xcite . recent advancements have targeted decoding strategies to address the off-target issue. @xcite used decoder pre-training and backtranslation to mitigate spurious correlations between language ids and outputs. @xcite introduced ebbs (ensemble with bi-level beam search), a two-level algorithm where ensemble components perform beam searches collaboratively, refining zero-shot translations. @xcite proposed source-contrastive and languagecontrastive decoding methods that identify probable translations by contrasting them with improbable alternatives, mitigating both hallucinations and off-target translations without retraining models. unlike prior work, our approach leverages a simple yet effective technique: using a single bilingual word or phrase pair as a fixed prefix for the source and target sentences. this strategy encour-ages llms to generate accurate translations from the outset while guiding them to consistently produce outputs in the target language. this method directly addresses the off-target problem with minimal resource requirements, making it particularly advantageous for low-resource scenarios. ## our approach we illustrate both the traditional zero-shot mt approach and our proposed method in figure 1 . at the top of figure 1 , the traditional zero-shot mt approach, referred to as "zero-shot" in the baseline, is depicted. this method relies solely on instructions without providing examples to guide the llms. for instance, to prompt an llm to translate a source sentence from english to italian, we provide a straightforward instruction, such as: "please translate the following sentence from english to italian:" alongside the given source sentence. the model then generates the expected italian translation for the given input sentence. to enhance zero-shot mt performance, one might add a full bilingual sentence pair, such as an english-italian pair example, to the prompting template. however, in practice, obtaining such bilingual sentence pairs is often difficult and laborintensive, requiring expertise from linguistic professionals, especially for low-resource languages. an alternative approach is to leverage available bilingual word or phrase pairs from a dictionary, using them as demonstrations. we refer to this method as the "one-shot" setting in the baseline in figure 1 . while this approach can provide some improvement, it may result in low-quality translations due to discrepancies between the example sentence and the given input sentence @xcite . to address this issue, we introduce our approach, illustrated at the bottom of figure 1 . in this method, we use a bilingual word or phrase pair {a, b} from a dictionary, where a represents a word or phrase in the source language, and b is its corresponding translation in the target language. the key idea is to select a such that it is neutral and relevant, making it suitable for adding to the beginning of most source sentences. this not only maintains the natural flow and semantic integrity of the original source sentence but also facilitates the generation of accurate target-language translations. for example, in figure 1 , we use the bilingual pair {a, b} as {as a result, di conseguenza}. therefore, the original source sentence "i achieved a good outcome in the examination" becomes "as a result, i achieved a good outcome in the examination.". similarly, the corresponding expected target sentence begins with di conseguenza, guiding the translation process effectively. ## conclusion this study introduces a new approach to enhancing zero-shot mt by employing fixed prefix pair bootstrapping. by strategically using a single bilingual word or phrase pair from a dictionary as a guiding prefix in both the source and target sentences, our method steers llms toward accurate translations while ensuring consistent generation in the target language. this targeted intervention effectively addresses off-target translation issues, a common challenge in zero-shot mt. extensive evaluations on the flores-101 devtest dataset across four language pairs and four different llms underscore the robustness and reliability of our approach, achieving consistent improvements over traditional zero-shot mt techniques.