A Study on the Effectiveness of Large Language Models for Translation with Markup (2023)

Abstract: In this paper we evaluate the utility of large language models (LLMs) for translation of text with markup in which the most important and challenging aspect is to correctly transfer markup tags while ensuring that the content, both, inside and outside tags is correctly translated. While LLMs have been shown to be effective for plain text translation, their effectiveness for structured document translation is not well understood. To this end, we experiment with BLOOM and BLOOMZ, which are open-source multilingual LLMs, using zero, one and few-shot prompting, and compare with a domain-specific in-house NMT system using a detag-and-project approach for markup tags. We observe that LLMs with in-context learning exhibit poorer translation quality compared to the domain-specific NMT system, however, they are effective in transferring markup tags, especially the large BLOOM model (176 billion parameters). This is further confirmed by our human evaluation which also reveals the types of errors of the different tag transfer techniques. While LLM-based approaches come with the risk of losing, hallucinating and corrupting tags, they excel at placing them correctly in the translation.

## introduction recent work involving large language models (llms) has shown impressive performance in various natural language processing (nlp) tasks. these models have the ability to perform few-shot (or in-context) learning based on prompts, an alternative to fine-tuning, requiring only a forward pass of the neural network @xcite . prompts are instructions in natural language given as input to llms along with a test sequence, allowing a few examples (i.e. few-shot) to be fed to the model at test time. researchers have shown that llms via prompting can be effective as machine translation (mt) systems @xcite @xcite @xcite , whose quality approaches that of traditional encoder-decoder neural mt (nmt) systems trained or fine-tuned on parallel corpora. the majority of the aforementioned research has been conducted on plain text, neglecting the practical application of mt for text containing markup, see in this paper, we conduct the first of its kind study on the use of llms for translation of text with markup where the transfer of markup tags, or tag placement, is as important as the translation of the content inside and outside the tags. we use sap's asian language dataset @xcite focusing on translation involving japanese, chinese, korean and english and experiment with zero, one and few-shot prompting of the open-source multilingual bloom and bloomz llms @xcite . we compare our results against those obtained via a general-domain mt system, m2m foot_0 @xcite , as well as a domain-specific in-house nmt system that handles markup tags via a detag-andproject approach. our multi-metric evaluations using bleu, chrf and comet reveal that while llms exhibit relatively poorer translation quality compared to the domain-specific nmt system, they are often competitive with a general-domain mt system, and that the degree to which llms are able to transfer markup tags out-of-the-box depends on the prompting strategy and the model size. this is further confirmed by our human evaluation that reveals the various error types associated with different tag transfer approaches. notably, the 176 billion parameter model employing few-shot prompting outperforms the detag-and-project strategy in terms of tag positioning, demonstrating its strong potential. our study focuses on the impact of example retrieval approaches, number of shots and their ordering. it provides insights for mt practitioners, and should encourage further research in this area. ## related work this paper focuses on an evaluation of llms for the translation of text with markup. we briefly review the related work in this area. @xcite present a data set from the it domain that features structure via inline markup, and corresponding mt results using a constrained beam search approach for decoding. further, @xcite compare different data augmentation methods with a detag-and-project approach, and evaluate on data from legal documents from the european union. the methods for tag transfer in @xcite are also related, even though they focus on inserting the tags into a fixed human translation. in contrast to these works, @xcite , who also release an evaluation dataset for structured document translation of asian languages, propose to use existing multilingual pre-trained nmt models as black-boxes for translating texts with inline elements directly. they show that these models perform surprisingly well at transferring markup tags during translation despite not being explicitly trained to handle structured content. in this paper, we further investigate black-box approaches for structured document translation, focusing specifically on llms. ## methodology the methodology employed in this work focuses on prompting approaches, namely, the template or format of instructions fed to the llms along with input sequences to be translated, as well as example retrieval techniques. ## 4 experimental setup in this section, we describe the datasets, language models and baselines used in our experiments to evaluate the utility of llms for structured document translation. ## results and analysis we now present our results for translation with markup for the experimental setup lined out in section 4. we provide a detailed analysis of the impact of various factors on the performance of llms on this task. a human evaluation will follow in section 6. ## human evaluation as the automatic lexical matching metrics used in this paper have their limitations in measuring mt quality @xcite , and evaluating tag placement automatically is a non-trivial task without a standardized methodology, we perform human evaluation to assess the correctness of translation and tag placement. we focus on the language pair english to japanese, for which translations of bloom 7b1 and bloom 176b both few-shot, m2m and the in-house nmt system (see ## conclusion we explored various llms and a specialized mt system to assess their ability to translate structured documents in the software documentation domain, focusing on both the translation quality and the transfer of markup elements. the investigation of different prompting approaches showed that llms learn from in-domain examples and are capable to produce correct text markup in the target language. with this respect, the foundation model bloom is more responsive to prompting than its fine-tuned variant bloomz. we also observed that the large-scale bloom model with few-shot prompting largely outperforms its smaller cousins in both translation quality and tag placement. however, this comes at a higher price and with subpar performance, which raises doubts about its practical usefulness for commercial translation purposes. while llms excel at transferring structural markup, most likely because they were trained on it, none of the investigated models achieve the translation accuracy of a dedicated machine translation system. nevertheless, this opens up interesting possibilities for future research, such as the combination of llms and mt systems to achieve the best of both worlds.