Exploring the Effectiveness of LLM Domain Adaptation for Business IT Machine Translation (2024)

Abstract: In this paper, we study the translation abilities of Large Language Models (LLMs) for business IT texts.We are strongly interested in domain adaptation of translation systems, which is essential for accurate and lexically appropriate translation of such texts.Among the open-source models evaluated in a zero- and few-shot setting, we find Llama-2 13B the most promising for domain-specific translation fine-tuning.We investigate the full range of adaptation techniques for LLMs: from prompting, over parameter-efficient fine-tuning to full fine-tuning, and compare to classic neural machine translation (MT) models trained internally at SAP.We provide guidance how to use training budget most effectively for different fine-tuning approaches.We observe that while LLMs can translate on-par with SAP’s MT models on general domain data, it is difficult to close the gap on SAP’s domain-specific data, even with extensive training and carefully curated data.

## introduction with swift improvement and recent successes of large language models (llms), it has become imperative for companies to measure their productive nlp systems against such new models. in the rapidly evolving field of nlp, incorporating the state-of-the-art models could unlock new capabilities for one's product and improve performance. on the other hand, switching to llm-based systems c 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. with this in mind, we set out to investigate whether llms can be easily utilized to outperform and ultimately supersede the current machine translation systems employed by sap @xcite . they are based on a traditional neural machine translation architecture trained on a multitude of data sources including the contents of the company-internal translation memories and is therefore optimized for sap's domain of interest, which we call business it here. while previous research has shown that llms make good translators @xcite @xcite , it is not yet well explored whether they can effectively adapt to domain-specific translation intricacies and outscore a smaller model that has been trained from scratch within the domain. in particular, our interest lays in whether comparably smaller sized open-source llms can be fine-tuned to this end. this interest is motivated by certain drawbacks of using large proprietary models such as openai's @xcite outof-the-box. potential data privacy concerns, slower inference and higher monetary costs (provided sufficient throughput) are some of the reasons. in addition, fine-tuning (open-source) models offers some more benefits. fine-tuning addresses challenges such as hallucinations and overgeneration, commonly associated with the llms' innate generative nature. by channelling the llms' focus towards translation through downstreaming, it becomes possible to regulate and control these unwanted generative tendencies, resulting in a more precise and tailored output for the intended domainspecific application. while there is a general argument to be made that fine-tuning an llm on parallel data would im-prove its translation quality, a much more crucial one can be made regarding domain-specific translation. in domain-specific translation, vocabulary and translation patterns might differ substantially from its general purpose counterpart. for instance, ui strings contained in our domain-specific data such as 'item masters in catalog', 'number in the total field' or 'pending the item has not yet been sent.' do not only differ from general domain texts in terms of vocabulary, but can also deviate substantially in syntax. in our experiments, we first estimate the translation capacities of various models through prompting to identify the most promising one to finetune. concretely, llama-2 13b appears to offer the best model size to performance trade-off. we then fine-tune the model with both full parameter and parameter-efficient tuning and conduct various ablation studies. overall, we arrive at the following conclusions: ## related work in their paper, @xcite presented the performance of gpt-3 and evaluated the translation ability of their llm. they found that generalpurpose llms benefit from having examples in the prompt (few-shot prompting) to guide the model towards a specific task. undoubtedly, adding relevant examples via few-shot prompting or retrieval augmented generation can improve translation performance. however, both alves et al. (2023) and li et al. (2023) observe that translation fine-tuning outperforms few-shot prompting when provided with only few thousands of training samples. xu et al. (2023) achieve state-of-the-art translation performance with help of a two-stage training mechanism, where in the first stage, the model is further pretrained on billions of tokens of monolingual data of various languages to shift the model to a more multilingually balanced state; away from its dominantly english pre-trained state. only then, the model is fine-tuned with limited parallel data. while the resulting performance is astonishing, the first training stage is computationally expensive. even with adequate gpu resources available, the proposed setup does not necessarily work as effective in lowresource domain translation, where the parallel data does not align well with the monolingual data used in the first stage. üstün et al. (2024) recently presented the aya model which uses a more balanced distribution of multilingual data in the pre-training. although the approach seems promising, our first preliminary investigations do not show substantially higher translation performance of the aya model compared to previous llms. ## datasets & evaluation in this paper, we mainly focus on the high-resource language pair @xmath0. the more resources a language pair has, the more llms should be able to leverage from their pre-training, making it easier and quicker to downstream them for the translation task. in addition, we also investigate the performance on the low-resource language pair @xmath1, which is known for its complexity, in section 5.7. for few-shot example retrieval, fine-tuning and testing, we use well-curated parallel sap-internal data. it is composed of large amounts of software user interface (ui) strings, user assistance (ua) texts, but also training materials, corporate content and marketing texts. the models are tested not only on a test set of 2000 segments consisting of domain-specific ui strings and ua texts, but also on general-domain data, i.e. the flores @xcite test set. even though they stem from the same domain (sap), training and in-domain test data are not merely divided randomly, but instead feature both temporal and distributional shifts. this in turn allows a more realistic performance evaluation @xcite . we want to clarify that sap's mt systems are trained using large corpora comprised of millions of parallel sentences, and this training is performed over many epochs. since fine-tuning llms on a similar scale would entail considerable computational costs, we conduct our fine-tuning experiments with fewer but gradually incremented quantities of data to map respective improvements in translation quality. furthermore, while sap's mt system evaluated in this study has been trained to excel in translating texts from the sap domain, it has not been fine-tuned to ui and ua texts specifically, and obviously the test data is unseen. we evaluate the performance with both bleu @xcite and comet foot_0 @xcite . while comet is more robust and correlates better with human annotators, the n-gram based bleu score nonetheless has its use when evaluating domain-specific translations. specifically, it captures lexical agreement with references which indicates the correct use of terminology and writing style for the domain. ## prompting to establish a baseline for the translation finetuning it is natural to start with simple prompt experiments. these experiments are relatively straightforward to conduct since large commercial models like gpt-4 are offered as services. furthermore, hosting open-source models for inference requires fewer resources compared to tuning them. the motivation driving this inquiry is twofold: first, to establish a baseline for the performance of opensource models; and second, to evaluate the inherent capabilities of gpt-4, a proprietary cutting-edge language model in its "out-of-the-box" state. given the nature of the task, we restrict ourselves to a selection of models that are intended for multilingual usage: figure 2 displays the bleu and comet scores of the models in the zero-shot setup. for opensource models, we establish a comparison under equal resource conditions, i.e. given four nvidia a10g foot_1 . models that are too large for full precision inference are tested with 8-bit quantization instead. as expected, sap's mt model performs best by a large margin on the domain-specific data. the superior bleu score, in particular, indicates the correct usage of in-domain vocabulary, rather than just semantically similar phrases. for general-domain data (flores), however, gpt-4 is capable of outperforming sap's domain-specific mt system out of the box. keep in mind that this mt model is not optimized for general-domain translation, and it is unknown whether the publicly available dataset was included in gpt-4's training data. provided with the simple prompt, all open-source models but llama-2 7b are able to translate texts consistently 3 , albeit not necessarily correctly. naturally, it is difficult to determine whether the weaker model performance is the result of shortcomings in its translation quality, or merely a misinterpretation or mishandling of the prompt. it might seem unintuitive, but some open-source models perform better on domain-specific data than on general-domain data. this can be explained by the large percentage of ui segments contained in the sap test set. while such strings contain lexical intricacies, they are generally short and syntactically simple, which makes them easier to translate for the smaller models. as a natural next step, we investigate whether few-shot prompting could narrow the gap for the sap domain-specific translations. we focus our experiments on gpt-4, serving as an upper llm benchmark, and llama-2 13b, which offered the best trade-off between model size and translation performance in the zero-shot experiments. in order to construct our few-shot prompts, we first encode the english source segments of the domain-specific parallel training data (section 3) with the sentence-bert model all-minilm-l6-v2 by reimers and 3 generating output in french and not english @xcite . then, for each english segment to translate, we retrieve the five translation pairs that have the highest cosine similarity to the english source. these pairs are then arranged as completed prompts and placed before the final segment set for translation, with each pair separated by an empty line. for the llama-2 13b model we had to conduct postprocessing of the output due to overgeneration issues. in particular, it continued generating english-french sentence pairs beyond the completed prompt. to deal with this, we simply truncated the generation after the first line break. the results of the few-shot experiments are displayed in while few-shot prompting can provide a basic understanding of the domain in question, it may not fully capture all the domain-specific nuances. this is particularly challenging when the domain is highly specialized, as selecting the appropriate domain-specific vocabulary and creating accurate examples can be difficult. additionally, including multiple examples in the prompt increases the token count, which can lead to higher computational costs and longer inference times. fine-tuning, on the other hand, would allow the model to learn from a much larger pool of examples, potentially leading to better adaptation to the specific-domain requirements. while the performance of open-source llms is promising, it falls short compared to both a domainspecific neural machine translation model as frequently used in production and more advanced models like gpt-4. a promising pathway to improve the task-specific performance of open-source llms is to further fine-tune them. given the substantial gap in performance on the in-domain data to sap's mt model and even gpt-4, we have a strong incentive to investigate how well llm translation can adapt to a specific domain through fine-tuning. to do so, we experiment with three different finetuning setups: 1. lora: with low rank adaptation @xcite , the pre-trained model weights are frozen while trainable rank decomposition matrices are injected on top of the frozen weight matrices. as the decomposition matrices are the only ones fine-tuned and contain magnitudes less parameters, model downstreaming becomes faster and more gpu efficient. 2. qlora: the fine-tuning approach proposed by @xcite quantizes the pretrained model during training and only keeps the trainable lora adapter weights in standard precision. this method reduces the memory requirements of fine-tuning which in turn, depending on the available gpus and model size, can allow data-parallel training rather than model-distributed one, cutting training time short by a multitude. 3. full fine-tuning: while @xcite and @xcite show that the proposed parameter efficient fine-tuning approaches perform on-par with full fine-tuning, other researchers applying them could not always con-firm such observations @xcite . consequently, we also conduct full fine-tuning to establish an upper bound. qlora is of special interest, as the greatly reduced gpu footprint allows cost-efficient training. in addition, quantization could also help reduce the cost during inference and recent development in dynamic adaptation @xcite ) make qlora even more tempting. a main interest of our experiments is therefore an evaluation of qlora against full fine-tuning for domain-specific translation. then, ablation studies are conducted that investigate shortcomings of qlora opposed to lora without quantization. ## conclusion we have shown that llama-2 13b shows great potential for domain-specific translation fine-tuning and can substantially improve over its zero-shot performance. however, doing so is no trivial task. we find that few-shot prompting is not sufficient to close the performance gap to productive systems in the sap domain. parameter efficient fine-tuning with low rank adaptation fails to internalize domainspecific phenomena and therefore cannot compete with a full fine-tuning. full fine-tuning, however, requires substantially more gpu compute power, which in turn is reflected in increased monetary costs. while we were able to approach the performance of the comparably small encoder-decoder mt sys-tem trained and employed at sap by continuously increasing training data, we were unable to surpass it. considering the much higher monetary inference costs and lower inference speed of the llm compared to the mt model, the benefit of switching systems is not immediately obvious, especially when separate models would be hosted for various languages. it is without doubt, however, that with rapidly improving released open-source models the performance for domain-specific llm translation finetuning is bound to increase as well. therefore, a continuous investigation into the translation capabilities of future open-source models is imperative.