MITRA -zh-eval: Using a Buddhist C hinese Language Evaluation Dataset to Assess Machine Translation and Evaluation Metrics (2025)

Abstract: With the advent of large language models, machine translation (MT) has become a widely used, but little understood, tool for accessing historical and multilingual texts. While models like GPT, Claude, and Deepseek increasingly enable translation of low-resource and ancient languages, critical questions remain about their evaluation, optimal model selection, and the value of domain-specific training and retrieval-augmented generation setups.While AI models like GPT, Claude, and Deepseek are improving translation capabilities for low-resource and ancient languages, researchers still face important questions about how to evaluate their performance, which models work best, and whether specialized training approaches provide meaningful improvements in translation quality.This study introduces a comprehensive evaluation dataset for Buddhist Chinese to English translation, comprising 2,662 bilingual data points from 32 texts that have been selected to represent the full breadth of the Chinese Buddhist canon.We evaluate various computational metrics of translation quality (BLEU, chrF, BLEURT, GEMBA) against expert annotations from five domain specialists who rated 182 machine-generated translations. Our analysis reveals that LLM-based GEMBA scoring shows the strongest correlation with human judgment, significantly outperforming traditional metrics. We then benchmark commercial models (GPT-4 Turbo, Claude 3.5, Gemini), open-source models (Gemma 2, Deepseek-r1), and a domain-specialized model (Gemma 2 Mitra) using GEMBA. Our results demonstrate that domain-specific training enables open-weights models to achieve competitive performance with commercial systems, while also showing that retrieval-augmented generation (RAG) significantly improves translation quality for the best performing commercial models.

## introduction evaluating machine translation (mt) systems remains a challenging endeavor, especially for literary contexts where a single "correct" translation is often elusive, and interpretation plays a significant role in determining quality. for many years, evaluation relied on string-similarity metrics such as bleu and chrf, which are not well suited for this scenario @xcite . however, the recent advent of deep learning-based methods has sparked a shift toward more sophisticated evaluation techniques, creating what some have aptly termed a "metrics maze" @xcite . although there are large-scale initiatives like the annual wmt evaluation campaign for high-resource languages, comparatively little attention has been devoted to assessing translation quality in literary, premodern, and low-resource domains. in this study, we address the unique challenges of assessing machine translation quality for premodern buddhist chinese into modern english, a task that involves bridging considerable cultural and temporal divides. for this, we introduce a novel dataset comprising 2,662 bilingual data points, carefully selected by domain experts to represent the full breadth of the chinese buddhist canon. additionally, we translate a subset of 182 data points using a range of machine translation systems and engage five domain experts to evaluate the quality of these translations. this not only allows us to measure inter-annotator agreement, but also to benchmark various automatic evaluation metrics against expert human judgment. subsequently, we assess both commercial and openweight machine translation systems on our dataset to provide an overview of the current performance landscape for this challenging language pair. finally, we conduct an ablation study to demonstrate how different data augmentation strategies can further enhance the performance of large language models (llms) in this specialized domain. our contributions can be summarized as follows: we make the datasets and evaluation pipeline used for this study available at https://github. com/dharmamitra/mitra-evaluation . ## related work so far, buddhist chinese has received little dedicated attention in nlp research. the first publication that trains and evaluates machine translation for this domain is @xcite , but they did not publicly release either their models or their training or evaluation datasets. another recent publication discusses the training and evaluation of machine translation systems for buddhist chinese @xcite . they released an evaluation dataset consisting of sections of a couple hundred sentence pairs taken from seven different texts. one detailed human-only evaluation compares the mt output of three buddhist texts from three llms (chat-gpt 4, ernie bot 4, and gemini advanced) @xcite . in the context of classical poetry, @xcite provides an evaluation benchmark for classical chinese poetical texts, which attempts to assess the poetic "elegance" of machine translations. more distantly related is @xcite , which examines how classical chinese to modern chinese data influences the process of historical korean document translation from hanja to modern korean and english. to summarize, in previous publications, the evaluation of machine translation performance for buddhist chinese has not played a main role. the only study that provides an evaluation dataset, @xcite , has only used sections from very few texts with very limited domain coverage. so far, there is no study assessing the quality of automatic metrics for machine translation evaluation for this idiom. ## dataset the evaluation dataset we present from buddhist chinese to english translation consists of 2,662 zh-en data points drawn from 32 chinese buddhist texts and their corresponding human translations. the chinese was taken from the cbeta corpus. 1 the translations were selected in a way such that they were distributed evenly across the canon to prevent bias towards certain sections. collectors were instructed to move in steps of fifty taishō foot_1 numbers and identify a translation close to either side of that number using the "bibliography of translations (by human translators) from the chinese buddhist canon into western languages" (bingenheimer ver 2024-11). 3 priority was given to translations that are not widely available online, e.g. the open-access translations published by bukkyō dendō kyōkai (仏教伝道協会), to mitigate the influence of data that is overrepresented in web-scraped datasets. 4 for each text, we collected the first 50-100 sentences after the prefaces and introductory paragraph. we cleaned lineend hyphenations, line returns, and deleted notes and note anchors. we use bertalign for sentencelevel alignment of the document pairs @xcite . while the oldest of the english translations date back to 1951, the majority was produced within the last 30 years, ensuring relatively consistent modern english usage across the reference translations this is the first balanced comprehensive evaluation dataset for buddhist chinese. crucially, it allows control for genre, i.e., it helps us understand whether the output quality is or is not dependent on the type of text that is translated. ## human evaluation and computed metrics in our evaluation of different metrics for this idiom, five human annotators independently assessed 182 machine-generated translations. these have been generated with machine translation systems of varying quality. we excluded any output of gemini 2 flash here, since this llm is also used as the judge for the gemba scoring, and evaluation of its own output could lead to undesired bias. all annotators hold phds or are doctoral candidates specializing in buddhist chinese texts. the annotators rated each translation on a scale from 1 (worst) to 5 (best), considering the source sentence, the machine translation output, and a reference translation. while we did not conduct specific annotation training, all evaluators worked with identical sets of sentences, allowing us to measure inter-annotator agreement. we evaluated several metrics against the human-annotated reference scores: bleu, @xcite , bleurt @xcite , chrf (character-level f-score) @xcite , and the llm-based gemba @xcite . for gemba, we implemented assessment using gemini 2.0 flash prompting on a scale of 0-100, and additionally tested a referencefree configuration (denoted as gemba*). we calculated both pearson and spearman correlations against each annotator's scores and present the averaged correlations in figure 1 . the results reveal weak average correlations for both bleu and chrf, supporting previous findings @xcite ) that these metrics are inadequate for evaluating machine translation output across different model types. while bleurt consistently outperforms bleu and chrf, both gemba variants demonstrate even stronger performance. notably, the reference-free gemba* achieves comparable spearman correlation to its reference-based counterpart, with only slightly lower pearson correlation. we attribute this performance pattern to potential issues in automatic sentence alignment and variations in human reference translation quality. based on these findings, we recommend using llm-based metrics, such as gemba, for evaluating buddhist chinese to english machine translation. particularly, reference-free llm-based evaluation proves highly effective, significantly outperforming traditional reference-based systems without needing to rely on costly manual data collection. ## model evaluation we compare the following different systems against each other: the commercial llms claude haiku 3.5 and claude sonnet 3.5, chatgpt 4 turbo, gemini 1.5 pro, as well as gemini 2 flash. these models were prompted between jan 15 and feb 10, 2025. we also evaluate the openly available llms deepseek-r1-distill-qwen-14b (deepseek-ai, 2025), gemma 2 9b it (team et al., 2024) as well as gemma 2 mitra, 5 which is based on gemma 2, but utilizes the buddhist chinese to english dataset presented in (nehrdich et al., 2023) together with additional domain-specific monolingual data in a continuous pretraining/fine-tuning setup (publication forthcoming). we further evaluate the two commercial llms gemini (ver. 2 flash and ver. 1.5 pro) as well as claude (3.5 sonnet) in a rag setup. with rag setup we mean a setup where the prompt of the llm is enriched by additional knowledge. in this case, this means retrieving relevant source-target sentence pair examples from bilingual data storage with a semantic embedding model and nearest neighbor search. a recent implementation of such a system that we take inspi-5 https://huggingface.co/buddhist-nlp/ gemma-2-mitra-it ration from is @xcite . in our case, we add @xmath0. our used prompt template is given in appendix a. we also compare different augmentation strategies for the rag setup in the ablation study. the averaged results per model are presented in figure 2 . the scores for each individual text are given in figure 3 . among all models, claude 3.5 sonnet rag shows the best performance, followed by gemini 2 flash rag. we acknowledge that since gemini 2 flash is also used as the judge in the gemba scoring system, the score might show bias in favor of this system. all the other major commercial llms gemini 1.5 pro, chatgpt 4 turbo, and claude 3.5 haiku show very similar performance across all texts with very similar overall trends. the open-source models, except for gemma 2 mitra, show a noticeable drop in performance. among these, deepseek qwen-14b is doing the best, at times matching the performance of the commercial llms. gemma 2 9b it is struggling to provide useful quality. the contrast in performance between this model and gemma 2 mitra shows that fine-tuning open-source models on an academic budget, even if their base performance is inferior, can lead to competitive performance with the right data selection. the rag setup improves the performance of both gemini 1.5 pro and claude, with the improvement being more pronounced in the case of gemini 1.5 pro. for both llms, the improvements are more pronounced for the earlier sections of the corpus, which are also better represented in the dataset. to identify significant changes in gemba score trends across documents, we applied a change point detection algorithm based on the pruned exact linear time (pelt) method @xcite with respect to the scores for all models, treating the documents as a time series. in our analysis, we set the penalty parameter (@xmath1. notably, the single change point detected with this penalty setting occurs after t1725, between categories 32 and 34: categories 1-32 comprise material presented traditionally as translated from indic sources into chinese. in contrast, categories 34 and onward consist of original compositions and commentaries composed in china that are not presented as direct translations from indian source texts. the bleurt scores too start to decline after t1725. the detected change point, combined with the lower values observed for categories 34 and higher, suggests that less suitable data is available to the llms for training on this type of data. this hints at a generally reduced translation activity and scholarly attention in texts not explicitly claimed to be based on in-dian originals. ## ablation study to investigate the impact of different data sources on rag translation performance for this evaluation dataset, we conducted an ablation study with the gemini-pro model across multiple configurations: for all augmentation settings, we used semantic embeddings and nearest neighbor search to retrieve a fixed number of 10 samples that are most closely relevant to the translation query segment. we show the results in ## conclusion and future work we have presented a comprehensive and balanced manually assembled dataset for the benchmarking of machine translation of buddhist chinese material into english. we further conducted a manual evaluation of automatically generated translations against their reference data, which enabled us to benchmark different evaluation scores, establishing gemba as the best-performing automatic evalation method. strikingly, we could show that the reference-free gemba* performs almost as good as reference-based gemba, which means that reliable evaluation of buddhist chinese to english machine translation is possible even when no dedicated reference data is collected. this is significant, since collecting domain-specific evaluation data is time-intensive and not many annotation experts exist who can do this type of work. we then conducted an evaluation of commercial as well as open-source llms on this dataset, mapping out the current performance landscape for this task. our results show that even highperforming commercial llms significantly benefit from data augmentation using curated domainspecific datasets, highlighting that dedicated data collection efforts are still crucial for optimal performance. the results also demonstrated that domainspecific fine-tuned models such as gemma 2 mitra vastly outperform other open-weight models and show competitive performance with commercial models, highlighting that fine-tuning such models can be very worthwhile for research communities. one research question was whether genre plays a role in translation performance. our experiments show no clear difference regarding the type of text. although the evaluation dataset is a cross-section of the canon, no genre stands out as particularly easy or difficult for current mt systems. the notable exception here is the divide between categories 1-32, which all models handle better, and 34 onwards, which all models handle worse, indicating that the autochthonous sections of the buddhist chinese canon are likely less represented in the training data of these models.