Evaluation of LLM for E nglish to H indi Legal Domain Machine Translation Systems (2025)

Abstract: The study critically examines various Machine Translation systems, particularly focusing on Large Language Models, using the WMT25 Legal Domain Test Suite for translating English into Hindi. It utilizes a dataset of 5,000 sentences designed to capture the complexity of legal texts, based on word frequency ranges from 5 to 54. Each frequency range contains 100 sentences, collectively forming a corpus that spans from simple legal terms to intricate legal provisions. Six metrics were used to evaluate the performance of the system: BLEU, METEOR, TER, CHRF++, BERTScore and COMET. The findings reveal diverse capabilities and limitations of LLM architectures in handling complex legal texts. Notably, Gemini-2.5-Pro, Claude-4 and ONLINE-B topped the performance charts in terms fo human evaluation, showcasing the potential of LLMs for nuanced trans- lation. Despite these advances, the study identified areas for further research, especially in improving robustness, reliability, and explainability for use in critical legal contexts. The study also supports the WMT25 subtask focused on evaluating weaknesses of large language models (LLMs). The dataset and related resources are publicly available at https://github.com/helloboyn/WMT25-TS.

## introduction machine translation (mt) has evolved from basic rule-based and statistical approaches to advanced neural network models, with recent advancements driven by large language models (llms) that utilize extensive pretraining datasets and transformer architectures. @xcite the legal domain poses significant challenges for mt, requiring precise handling of context-dependent terminology @xcite , complex sentence structures, and the accurate conveyance of cultural and jurisdictional nuances due to varying legal systems. this complexity surpasses that found in general language translation, making high levels of lexical accuracy, logical coherence, and syntactic fidelity essential for effective legal translation. the text highlights the crucial importance of accuracy and fidelity in legal document translation due to its high-stakes nature. it emphasizes that even minor mistranslations can lead to serious legal and financial consequences such as contractual disputes and judicial errors. therefore, precise legal translation is essential to support international legal cooperation, manage cross-border litigation, provide equitable access to justice for non-native speakers, and make legal information accessible to a wider audience.(wmt) 1 the series has consistently served as a pivotal platform, instrumental in benchmarking progress and driving innovation within the machine translation research community across a diverse array of language pairs and domains. wmt25 @xcite continues this vital tradition, offering meticulously designed specialized test suites that push the boundaries of current mt technologies and identify areas for future breakthroughs. this paper specifically focuses on the wmt25 legal domain test suite for english to hindi foot_1 , embarking on an in-depth investigation into how various llm-based mt systems perform when compared to more traditional and established hybrid approaches. our overarching objective is to provide a comprehensive and nuanced analysis of their efficacy in this demanding domain, meticulously identifying the top performing contenders and critically discussing the broader implications of our findings for the future trajectory and practical application of legal machine translation systems, including considerations for deployment and ethical use. ## related work the wmt shared tasks have consistently been a primary driving force behind significant advancements in machine translation research, fostering innovation and providing a standardized, competitive benchmark for evaluating system performance @xcite . previous wmt editions, notably those from wmt24 (as evidenced by a series of influential papers such as @xcite to ( ármannsson et al., 2024) ), have unequivocally showcased the increasing dominance and sophistication of neural mt (nmt) models @xcite . these works have meticulously detailed a wide array of architectural innovations, including the widespread adoption of transformer networks, advanced training methodologies such as back-translation and knowledge distillation, and impressive performance gains across diverse language pairs and specialized domains. key themes emerging from this extensive body of research include the paramount importance of largescale pre-training on vast textual corpora to learn robust linguistic representations, the efficacy of fine-tuning models on domain-specific data @xcite to enhance specialized vocabulary and stylistic nuances (e.g., legal jargon, formal tone), and the continuous refinement of robust evaluation metrics to more accurately reflect human judgment of translation quality. techniques like data augmentation (e.g., synthetic data generation), transfer learning @xcite ) from high-resource to lowresource languages, and the development of more efficient attention mechanisms have been central to these advancements, enabling nmt models (?) to capture intricate linguistic patterns and contextual dependencies with greater precision than their predecessors. the rapid advancement of large language models (llms) like gpt, gemini, and claude has significantly transformed machine translation (mt) research by challenging existing paradigms. these models, originally crafted for general language tasks, have shown impressive zero-shot and fewshot translation skills due to their training on vast, diverse datasets. they excel in capturing complex semantics and context, making them promising for specialized fields such as legal translation, where precision and adherence to terminology are critical. research is actively exploring their @xcite adaptation for specific mt tasks, and it often outperforms traditional neural machine translation (nmt) @xcite models in challenging situations, such as low-resource languages and complex linguistic features @xcite . nonetheless, adapting llms to specific domains poses challenges, such as the risk of losing general linguistic knowledge, generating plausible but incorrect legal outputs, and maintaining strict legal fidelity without creative rephrasing. ## results and observation the performance of the systems evaluated on the wmt25 legal domain test suite (english to hindi) is meticulously summarized below, directly derived from the provided evaluation results: ## conclusion the wmt25 legal domain test suite for english to hindi machine translation highlights significant progress made by large language models (llms) in specialized domain translation. notably, gemini-2.5-pro excels, outperforming others across multiple evaluation metrics and emphasizing the potential of advanced llm architectures with domain-specific pre-training or fine-tuning. these models effectively handle complex legal language and structures, showcasing their sophisticated linguistic capabilities. while llms show dominance, traditional and hybrid mt systems also demonstrate competitiveness, indicating their continued relevance. the study underscores the importance of model scale, architecture, and domain adaptation for success in legal mt. it suggests that llms will play an increasingly central role in legal translation, advancing accuracy and efficiency. however, ongoing innovations across mt paradigms are needed to balance performance with reliability and ethical considerations, given the high stakes of errors in the legal domain.