Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles (2024)

Abstract: Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. However, even though zero-shot translations are relatively good, there remains a discernible gap comparing their performance with the few-shot setting. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus. Additionally, we explore potential approaches to enhance zero-shot baselines without the need for parallel demonstration examples, providing valuable insights into how these methods contribute to improving translation metrics.

## introduction recent advancements in large language models (llms) have revolutionized natural language processing field as such models @xcite @xcite can easily adapt to a new task through prompt (in-context) learning where the task instruction and demonstrations (examples to guide llms on the task) are provided to the model. such capability opens up new opportunities for machine translation, which is traditionally trained/fine-tuned on large amounts of parallel corpus @xcite @xcite . recent work @xcite @xcite has found that prompt-based methods perform well on language models trained primarily on monolingual data, rivaling state-of-the-art systems trained specifically for machine translation tasks on benchmark datasets. comparing zero-and few-shot outputs ( §2), we observe a huge gap in terms of bleu (papineni source die persönliche haftung der bediensteten gegenüber der beobachtungsstelle bestimmt sich nach den fü das personal der beobachtungsstelle geltenden vorschriften. ## the gap between zero-and few-shot for our experiments, we use the german-english data splits in aharoni and goldberg (2020) 2 that comprises it, subtitle, law, medical, and koran domains. due to the data quality issue, we keep law, medical, and koran domains as they are relatively cleaner and representative of specialized domains other than generic daily language. for the details of setups and discussion about the dataset, we defer readers to appendix a. based on prior work on in-context learning @xcite @xcite @xcite , inter alia), we design prompts to instruct llms to perform zero-and few-shot translations. specifically, for few-shot translation, we employ a retrieval-based prompting method @xcite @xcite that retrieves source and target pairs from the training corpus as in-context learning (icl) demonstrations. to facilitate this process, we create a prompt template illustrated in figure 2 left. in this 2 the dataset is originally collected by @xcite . ## how much does style contribute? we have observed that the lexical difference between zero-shot and few-shot translations is more noticeable than their semantic difference. following the traditional linguistic definition of language style @xcite , inter alia), we can interpret this phenomenon as a distinction in writing style, where semantics remains consistent while aspects such as word choices, syntax, tones, etc., are altered. to examine the potential impact of writing styles being learned through few-shot demonstrations, we investigate various aspects associated with writing styles to determine whether few-shot translation exhibits more consistent styles with target domains. we follow common techniques used in prior work @xcite @xcite to analyze style matching through three aspects: lexical overlaps, syntactical structures, and 3 these examples serve as reference points to help readers qualitatively grasp the disparity in comet scores. word dependencies. these aspects provide valuable insights into the output style of different systems. ## narrowing the gap via matching styles from §3, we demonstrated that few-shot conforms to the target style better than zero-shot translation. motivated by this observation, we investigate approaches to mitigate the style difference so that zero-shot translation can be improved without access to additional parallel examples. we propose a novel style-learning prompting approach that leverages target corpora, as illustrated in figure 2 right. our hypothesis is that style-relevant information primarily resides in the in-domain target corpus (in our case, english). therefore, by retrieving samples from the corresponding monolingual corpus, the style-learning instruction would enhance style consistency in translation. this idea is in the same spirit as prior work in test time domain adaptation @xcite @xcite , where target domain information is used to guide model inference. in our case, we allow language models (llms) to access examples from the target corpora, enabling them to adjust their language styles. our approach has two steps: first, we employ zero-shot translation; then we take the translation output as a query to retrieve examples from the target language's corpus and add them to the original translation prompt with a style-learning instruction. by revising prompts with target examples, the approach allows us to align styles in translations without source-side demonstrations. the performance comparison foot_5 among few-shot, zero-shot, and zero-shot with style matching is shown in figure 4 , where the number of demonstrations ranges from 1 to 50. from the results, few-shot prompting exhibits the best performance by leveraging both source and target information. our style-learning prompts largely improve the zero-shot performance, reducing the gap by approximately 70%. to further validate these findings, we conducted experiments using the llama2-7b model @xcite , fixing the number of demonstrations at 5 for both german-english and english-german translation directions. as depicted in despite the remaining performance gap, our style-learning prompting method demonstrates increased data efficiency and adaptability of the language model to new domains where only monolingual resources are available. ablation on retrieval quality for both fewshot and our style-learning prompting, the quality of retrieved examples is important. we retrieved the 100 most similar samples with bm25 and separated them into four groups (tier 0 to 3, with 0 being the most similar group). as shown in ## conclusion in this paper, we investigate the prompt-based zeroand few-shot translation performance on different domains. we attribute the performance gap to the style difference, and through our analysis, we verified that few-shot translation benefits from its retrieved samples by learning the target domain's style. we then design a style-learning prompting strategy that only requires monolingual information and effectively bridges the gap between zero-and few-shot translation.