How Effective is Synthetic Data and Instruction Fine-tuning for Translation with Markup using LLM s? (2024)

Abstract: Recent works have shown that prompting large language models (LLMs) is effective for translation with markup where LLMs can simultaneously transfer markup tags while ensuring that the content, both inside and outside tag pairs is correctly translated. However, these works make a rather unrealistic assumption of the existence of high-quality parallel sentences with markup for prompting. Furthermore, the impact of instruction fine-tuning (IFT) in this setting is unknown. In this paper, we provide a study, the first of its kind, focusing on the effectiveness of synthetically created markup data and IFT for translation with markup using LLMs. We focus on translation from English to five European languages, German, French, Dutch, Finnish and Russian, where we show that regardless of few-shot prompting or IFT, synthetic data created via word alignments, while leading to inferior markup transfer compared to using original data with markups, does not negatively impact the translation quality. Furthermore, IFT mainly impacts the translation quality compared to few-shot prompting and has slightly better markup transfer capabilities than the latter. We hope our work will help practitioners make effective decisions on modeling choices for LLM based translation with markup.

## introduction while a significant majority of machine translation (mt) research has been conducted on translating plain sentences from one language to another, much of the web and proprietary or business documents requiring translation come in structured formats like html pages or microsoft office files containing markup. therefore, practical mt systems should be adept not only at translating plain sentences but also sentences with markup (see figure 1 for an example), where the task is to translate content in the source language while simultaneously ensuring that markup tags wrap the appropriate content in the target language. until the advent of deep learning, the most commonly used approach for handling markup was the detag-and-project approach @xcite , which is not end-to-end and is prone to error compounding from individual components such as the mt system, word-aligner and projection algorithms. therefore, using end-to-end neural networks for translation with markup @xcite makes a more attractive solution. recently, researchers have shown that transformer @xcite based large language models (llms) @xcite can seamlessly translate sentences with markup despite not explicitly being trained to do so @xcite . they show that few-shot prompting @xcite enables llms to transfer markup tags when translating from source to target languages. surprisingly, despite being general purpose, their markup transfer capabilities approach, if not surpass, highly optimized models figure 1 : examples with inline markup, inspired by @xcite . trained specifically for this purpose. however, they assume the existence of high-quality parallel corpora with markup when prompting, and this kind of data may not always be available. furthermore, while they utilize pre-existing generic instruction fine-tuned (ift) models, they do not ift their own mt models, the effectiveness of which remains unknown. in this paper, we fill this gap via a twopronged exploration on the effectiveness of synthetic data and ift for translation with markup. we take the case of translation from english to five european languages, german, french, dutch, finnish and russian, and first establish the efficacy of zero-and few-shot prompting on a popular open-source llm, namely bloom (le @xcite . following this, we explore approaches for synthetically creating parallel data with markup to understand its efficacy for prompting. we further deepen our investigation by performing ift of bloom with both clean and synthetic data and attempt to discern settings in which synthetic data can be useful. we show that regardless of few-shot prompting or ift, synthetic data created via word alignments leads to slightly inferior markup transfer compared to high-quality human-curated data; however, it does not negatively impact the translation quality. furthermore, somewhat surprisingly, we find that ift itself mainly improves the translation quality compared to few-shot prompting and has only slightly better markup transfer capabilities than the latter. we hope our findings will act as guidelines for practitioners to make effective decisions on modeling choices for translation with markup. ## related work our work focuses on machine translation with markup, llms and synthetic data. ## methodology figure 2 presents an overview of the methodology followed in this paper: few-shot prompting in section 3.1, instruction fine-tuning (ift) in section 3.2, and our methods for creating synthetic parallel data with markup in section 3.3. ## experimental settings this section describes datasets, implementation details, and various settings for analysis. ## results we structure the results in two major sections: the first focusing on synthetic data and using it for prompting, and the second focusing on ift along with synthetic data. ## conclusion in this paper, we have studied the effectiveness of synthetic data and instruction fine-tuning for translation with markup. we observed that an llm without few-shot prompting or ift already has impressive markup transfer capabilities, but suffers from low translation ability in the document domain. although few-shot prompting can help improve translation quality, ift is more effective, while also improving markup transfer capabilities regardless of whether high-quality or synthetic data was used. in the future, we would like to explore more controllable and scalable ways to generate synthetic data and eliminate the need for human curated data. zenkel, t., wuebker, j., and denero, j. (2021). automatic bilingual markup transfer. in findings of the association for computational linguistics: emnlp 2021, pages 3524-3533.