How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes (2024)

Abstract: In this study, we explore the effectiveness of fine-tuning Large Language Models (LLMs), particularly Llama 3 8B Instruct, using translation memories (TMs) for hyper-specific machine translation (MT) tasks. Decoder-only LLMs have shown impressive performance in MT due to their ability to learn from extensive datasets and generate high quality translations. However, LLMs often struggle with the nuances and style required for organisation-specific translation so we leverage TMs, which store human translated segments, as a valuable resource to enhance translation accuracy and efficiency. We investigate the impact of fine-tuning the Llama 3 model using TMs from a specific organisation in the software sector. Our experiments cover five translation directions across languages of varying resource levels (English to Brazilian Portuguese, Czech, German, Finnish, and Korean). We analyse diverse sizes of training datasets (1k to 100k+ segments) to evaluate their influence on translation quality. We fine-tune separate models for each training set and evaluate their performance based on automatic metrics, BLEU, chrF++, TER, and COMET. Our findings reveal improvement in translation performance with larger datasets across all metrics. On average, BLEU and COMET scores increase by 13 and 25 points respectively on the largest training set against the baseline model. Notably, there is a performance deterioration in comparison with the baseline model when fine-tuning on only 1k and 2k examples; however, we observe a substantial improvement as the training dataset size increases. The study highlights the potential of integrating TMs with LLMs to create bespoke translation models tailored to the specific needs of businesses, therefore enhancing translation quality and reducing turn-around times. This approach offers a valuable insight for organisations seeking to leverage TMs and LLMs for optimal translation outcomes, specially in narrower domains.

## introduction in recent years, decoder-only large language models (llms) have revolutionised the machine transla-tion (mt) field due to their ability to learn from vast amounts of data and generate high-quality translations @xcite @xcite @xcite ## results and discussion the results in to provide a point of comparison, we evaluate the performance of gpt-3.5 foot_8 on our test set. while gpt-3.5 outperforms our highest-performing model in bleu and chrf++ for de and fi, the 100k+ datasets often surpass gpt-3.5 in other languages and metrics. this demonstrates the effectiveness of creating bespoke models through fine-tuning midsized llms when leveraging domain-specific data. targeted fine-tuning can yield competitive or superior results compared to larger, general-purpose models like gpt-3.5. ## conclusions fine-tuning on tms has been demonstrated to enhance the performance of llms in mt tasks. in this paper, we investigate the relationship between automatic metric results and training set sizes to identify the optimal balance where resource investment yields the most significant improvements in translation quality. in our experiments, it has become evident that fine-tuning on training datasets whose size is larger than 5k examples returned increasingly better results in 19 out of the 20 language-training set size combinations explored. by leveraging tms, the model becomes more adept at recognising and reproducing previously translated segments, their style, and terminology. furthermore, fine-tuning on tm data helps the model adapt to specialised fields. the test and training sets used come from a much narrower corpus of data than in similar experiments that deal with wider domains, i.e. medicine @xcite . the hyper-specific nature of the training data employed in our approach may partly explain the promising results. we therefore leverage the advantage that smaller models licensed for business-use offer; they can be adapted several times over for narrow and specific domains, as well as multiple languages with little investment, instead of aiming for a more general purpose or multilingual model. the hyper-specific purpose of our trained model, i.e. one language direction and a narrow domain, suits the size and easiness of training of an 8b parameter mode. being a commonly experienced scenario in the localization industry, this is an under-explored approach that organisations could be pursuing in order to make the most out of their access to tms and llms for mt in order to obtain the best possible re-turn on investment when leveraging their previously human-translated material. low-resource languages seem to be in a perfect position to benefit from leveraging small businessfriendly models, like llama 3 8b. the gains in automatic metric results for ko are substantially higher for high resource languages, like pt-br and de, returning the highest increase in performance compared to the metrics obtained from training on similar set sizes in those languages. ko observes an increase of 130% on comet from the baseline to the 100k+ dataset, whereas the average increase amongst the other target languages is 46% (cf. it is important to mention that, just as wu (2024) acknowledges the flores-200 dataset leakage into llama 3, it is possible that some of our test set was also scraped by the llama 3 models, as parts of the material were published online prior to the llama 3 family's pre-training. we face the same challenge as the whole ai researching community, forced to either constantly come up with new test sets or simply acknowledge the potential leakage of test data @xcite . we urge large tech companies to disclose at a minimum the test sets that were not ingested by their models for the benefit of the whole community. we acknowledge the llama team's leadership in this area @xcite .