Domain Dynamics: Evaluating Large Language Models in E nglish- H indi Translation (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in machine translation, leveraging extensive pre-training on vast amounts of data. However, this generalist training often overlooks domain-specific nuances, leading to potential difficulties when translating specialized texts. In this study, we present a multi-domain test suite, collated from previously published datasets, designed to challenge and evaluate the translation abilities of LLMs. The test suite encompasses diverse domains such as judicial, education, literature (specifically religious texts), and noisy user-generated content from online product reviews and forums like Reddit. Each domain consists of approximately 250-300 sentences, carefully curated and randomized in the final compilation. This English-to-Hindi dataset aims to evaluate and expose the limitations of LLM-based translation systems, offering valuable insights into areas requiring further research and development. We have submitted the dataset to WMT24 Break the LLM

## introduction machine translation (mt) @xcite has witnessed significant advancements with the advent of large language models (llms) @xcite , which leverage extensive pretraining on massive datasets to achieve high performance across various language pairs @xcite @xcite . despite their remarkable generalization capabilities, llms often struggle with domain-specific texts due to a lack of targeted training on such specialized content @xcite @xcite . some llms @xcite generate good translation involving low-resource language when target language is english but not the other way around @xcite . these challenges are amplified when the domains involved are different from those of training data. this limitation poses a challenge for deploying mt systems in real-world applications where domainspecific accuracy is crucial. to address this gap, we have collated this dataset that exposes the difficulties faced by llm-based mt systems when dealing with domain-specific content. we have combined sentences from judicial, educational, religious, literature, and noisy user-generated content domains. each domain-specific subset comprises approximately 250-300 sentences, which are then randomized to form the final dataset. this dataset, focusing on the english-to-hindi translation direction, aims to rigorously test the robustness and adaptability of llm-based mt systems. by identifying the translation challenges specific to each domain, our study provides valuable insights for improving domain adaptation techniques in machine translation, ultimately contributing to more reliable and accurate mt solutions for specialized applications. our contributions to the paper are as follows: ## related works neural machine translation (nmt) has made significant progress, especially for high-resource languages, but translating low-resource languages remains a challenge. for example, the translation of indic languages like hindi is difficult due to the scarcity of high-quality parallel corpora. multilingual models like indictrans @xcite and indictrans2 @xcite show performance improvements, yet domain-specific performance data is lacking. for domain-specific augmentation, @xcite used pre-trained language models to generate synthetic in-domain data through back translation for arabic-english translation. in the lowresource context, gain et al. (2022) explored english-hindi translation in chat-based conversations, while @xcite introduced the edumt dataset to enhance english-hindi translations for educational content. domain adaptation techniques have also been applied for specialized translations, such as chemistry related and general english-hindi texts @xcite . in the legal domain, recent studies like briva-iglesias et al. @xcite show that llms outperform google translate for legal texts, and @xcite developed a custom dataset for english-nepali legal translation. for the literary domain, nmt has been applied to german-english (matusov, 2019), english-slovene @xcite , and english-turkish @xcite translations, with mixed results on automatic versus human evaluation @xcite . noise robustness in nmt is critical, as noisy inputs can degrade translation quality. studies like @xcite explored noise effects, while @xcite introduced the mtnt dataset. recent efforts used llms to filter noise and enhance nmt performance @xcite . finally, nmt has also been applied to e-commerce, particularly to translate product reviews. gupta et al. (2022) utilized transfer learning from english-kannada, english-gujarati, and english-marathi models for lambani, a low-resource tribal language. additionally, they examined the impact of freezing specific encoder and decoder layers during training. ## dataset large language models (llms) excel in general machine translation but struggle with specialized domains. our dataset includes english-hindi bitext pairs from six critical domains, aiming to improve llms' translation accuracy in these areas, which is vital for advancing their capabilities. ## evaluation in this section, we outline the various evaluation techniques employed to assess the performance of the models based on their outputs. the evaluation metrics considered in this study are the bleu @xcite score, comet @xcite score, and human evaluation score. ## conclusion this paper presents a comparison of various model submissions for the wmt shared task 2024. we proposed a dataset with domain-wise segregation and conducted a domain-specific analysis of the submitted models. our comprehensive evaluation using bleu, comet, and human assessments of the machine-translated hypotheses identified claude 3.5, transmissionmt, unbabel tower 70b, online-a, and online-b as some of the topperforming models for machine translation using llms. the analysis revealed that the formal do-mains of general and education are the easiest for models to handle, whereas the noisy and review domains proved to be the most challenging. this study highlights that while llms show proficiency in machine translation, there is still significant room for improvement.