MITRA -zh: An efficient, open machine translation solution for Buddhist C hinese (2023)

Abstract: Buddhist Classical Chinese is a challenging low-resource language that has not yet received much dedicated attention in NLP research. Standard commercial machine translation software performs poorly on this idiom. In order to address this gap, we present a novel dataset of 209,454 bitext pairs for the training and 2.300 manually curated and corrected bitext pairs for the evaluation of machine translation models. We finetune a number of encoder-decoder models on this dataset and compare their performance against commercial models. We show that our best fine-tuned model outperforms the currently available commercial solutions by a considerable margin while being much more cost-efficient and faster in deployment. This is especially important for digital humanities, where large amounts of data need to be processed efficiently for corpus-level operations such as topic modeling or semantic search. We also show that the commercial chat system GPT4 is surprisingly strong on this task, at times reaching comparable performance to our finetuned model and clearly outperforming standard machine translation providers. We provide a limited case study where we examine the performance of selected different machine translation models on a number of Buddhist Chinese passages in order to demonstrate what level of quality these models reach at the moment.

## introduction regarding the languages of the buddhist tradition, there is a striking gap between the amount of available material in their ancient source languages pāli, sanskrit, buddhist classical chinese, and tibetan, and the number of available translations into western languages, leaving the majority of texts inaccessible to a wider audience. in the case of buddhist chinese, only about 10% of the digitally available material has been translated into western languages over the last two centuries. machine translation (mt) therefore holds a great promise to help in the process of translating these texts, which is proceeding at a slow pace so far. translators of buddhist chinese texts into western languages generally do not work with mt tools yet, as their performance is rather poor. for common tasks in digital humanities such as topic modeling or semantic comparison, mt models also hold great promise as they make it possible to apply models trained primarily on english to this material. on the technical level, recent years have brought substantial advances in the performance of data-efficient mt systems, and their high level of reliability for language directions with good resources such as french or german to english has led to their wide adoption. in the case of low-resource languages with only limited data resources, the training of stable and usable mt systems remains difficult. this paper deals with the challenging situation of buddhist classical chinese, which has been generally neglected in the compilation of openly available large parallel datasets such as opus foot_0 or the training of multilingual mt models such as nllb @xcite . we make the following contributions to this problem: in section 2 we give an overview of the relevant literature. section 3 describes the datasets as well as the two augmentation strategies. in section 4 we show the evaluation results of the different models. in section 5 we examine the performance on different domains and conduct the case study. ## related work along with natural language processing in general, the field of neural mt was revolutionized by the introduction of the transformer architecture in 2017 @xcite . these networks introduced a way of processing language that emphasized determining which parts of a sentence should be attended to. by processing a number of tokens simultaneously (rather than sequentially), transformers are capable of learning relations between more distant parts of a sentence than had been possible in widely used models like lstms. the resulting revolution in nlp is ongoing, with current iterations of models like openai's chatgpt hardly needing any introduction. transformers come in 3 broad flavors: encoder-only, decoder-only, and encoderdecoder. decoder-only models such as gpt are trained to predict the next token in a partially completed sequence, while encoderdecoder models learn to encode inputs and then decode them appropriately. in this paper, we will use decoder-only and encoder-decoder variants, with mbart50 @xcite , wmt21 @xcite and nllb @xcite being encoderdecoder models while llama2 is decoder-only. the first of these, mbart50, utilizes monolingual pretraining with a denoising objective. nllb on the other hand is trained directly on a massive multilingual parallel dataset without a pretraining stage. the wmt21 model we use consists of dense english to many and many to english models for translating between various languages (including chinese). the open-source llama2 @xcite is a decoder-only models which is pretrained on massive monolingual datasets consisting primarily of english data. we decided to work with mbart and nllb since both models have shown significant jumps in performance on low-resource languages when compared to randomly initialized transformer configurations. we will not use mt5 @xcite , which follows a similar pretraining objective as mbart, in our evaluation scheme since mbart has shown to be of equal or slightly better performance on lowresource translation tasks @xcite . the first and to our knowledge only dedicated publication on the problem of buddhist classical chinese mt is @xcite . unfortunately, they have not made their models or evaluation data available, making it impossible to compare their findings in this paper. ## dataset we collect a total number of 209,454 bitext pairs/5,738,025 characters from a variety of texts of the taishō canon that have been translated into english. due to the mechanics of the tokenizers of common language models, a single chinese character is roughly equal to one token. the detailed genre distribution is given at table ??. the composition dates of the chinese texts in this dataset range from about ca. 150 to 1600 ce, while the english translations have been composed between ca. 1900 and 2020 ce. the training dataset covers a wide variety of different buddhist chinese domains: early and mahāyāna sūtras, canon law, philosophical treatises, commentaries, ritual texts, etc. about half of the texts are translations of indic buddhist sources, others were composed directly in buddhist classical chinese. due to the diachronic spread and the as a first step, the english translations have been digitized and optical character recognition has been applied when necessary in order to obtain machine-readable unicode text. the translations have then been aligned with their chinese counterparts as contained in the chinese buddhist electronic text association (cbeta) corpus. 3 many cbeta texts are based on an early 20th-century canonical edition, the "taishō canon". the texts have been thoroughly proofed against their original print editions. in some cases, punctuation has been added which increases intelligibility for humans (at least). since on average the language models that we are evaluating have been exposed to more training data in simplified cjkv characters than in traditional cjkv, we convert the characters in the dataset to simplified characters during preprocessing. the alignment was performed with vecalign @xcite . the embedding model used for the alignment process is a modified version of the multilingual sentence embedding model labse @xcite . this model was further finetuned on a small corpus of gold-quality buddhist chinese to english bitext pairs. in order to reduce the influence of misaligned sentences, we use a rule-based scheme to remove sentences where the aligned english section is either much shorter or much longer than the chinese counterpart. we also exclude samples from the training process where the chinese part is shorter than four characters, assuming that nmt models do not learn well from very short samples. we manually curated an evaluation dataset with a total size of 2,300 bitexts from a variety of texts from different genres of the buddhist chinese canon. passages of the following texts have been included: t0026 (447 bitexts), t0374 (518 bitexts), t0475 (185 bitexts), t1585 (307 bitexts), t1600 (784 bitexts), t1970 (234 bitexts), t2062 (66 bitexts). the alignment of the evaluation dataset was performed by vecalign and then manually postcorrected. training and evaluation data can be made available on request. ## experiments we evaluate the following models: bing translator 7 , deepl 8 and google translate 9 are commercial translation engines. we test chatgpt3.5 and gpt4 are the commercial chat systems provided by openai. we query the openai models with a simple prompt: "translate the following buddhist chinese passage into english: <@xmath0:". transformer 600m serves as the baseline for the finetuned models, which is the nllb600m model with randomly initialized weights, simulating the training of a transformer model with 600m parameters from scratch. mbart50 and mbart50-to-1 are two different versions of the multilingual bart model with a size of 611m parameters @xcite . both are pretrained on a denoising task, while the latter is the many-to-one version that is finetuned on a many-to-one translation task, including chinese, with english as the target language. no further information is provided during the prompting step. nllb600m-3.3b is the massive multilingual model of meta ai in different sizes, trained among other languages also on chinese to english. wmt21 is the meta ai's submission to wmt21 news translation task @xcite . we use the wmt21dense-24-wide-x-en version with 4.7b parameters, which was also trained on the chinese to english task. we fine-tuned the 7b parameter version of llama2 available on huggingface, using qlora for fintetuning on the full parallel 7 https://www.bing.com/translator 8 https://www.deepl.com/translator 9 https://translate.google.com/ dataset. during training and inference, we used the prompt "below is some text in classical chinese. it is taken from the buddhist literature. write a translation of the text into english." followed by labelled chinese inputs and a label for the english translation. we finetune all encoder-decoder models on sentence-level and on pseudo-paragraph-level as we assume that a larger context might help the models to arrive at better translation solutions. for the pseudeo-paragraph level, we concatenate adjecent sentences with a total length of up to 200 tokens. we decided on this number as the encoder-decoder model with the shortest context length, wmt21, only supports up to 200 tokens. we did a thorough hyperparameter search on a fixed holdout set to determine the optimal learning rate and number of training steps for each model. ## analysis in order to understand the nature of the mistakes that the different translation models produce, we analyzed several passages manually. we give the full samples in the appendix. the first paragraph is taken from t1585, the cheng weishi lun, a core text of sino-indian yogācāra philosophy. on this text, wmt21+aug-ft shows a generally superior quality, producing less serious mistakes, which mirrors the bleu score results on the individual texts. it is noteworthy that while all three models on average use the right vocabulary to translate the philosophical terms in this paragraph, chatgpt3.5 struggles significantly and gpt4 struggles somewhat to interpret the dense syntax of the chinese. wmt21+aug-ft is doing visibly better, but certain points of confusion remain, i.e. rendering 非無 (here: "not nonexistent") as "neither nonexistent nor existent", which is not correct. for t0026, an early buddhist sūtra text, the bleu and chrf++ score does not well align with our manual evaluation. although the metric indicates a clear advantage for wmt21+aug-ft, many passages are actually rendered more accurately in the gpt4 output. it is possible that the metric was influenced by the tendency of wmt21+aug-ft to use sanskrit terms for their chinese equivalents, something that is common practice in buddhist translation. in our example, the five great rivers of india (jambudvīpa) are mentioned and while the gptx models render 恒 伽， 搖 尤 那， 舍 牢 浮， 阿 夷 羅 婆 提， 摩 企 with at times misleading pinyin transcriptions (hengqie/hengqia, shalao fu/sheloufu etc.), wmt has "ganges, the yamunā, the śrāvastī, the ajiravatī, and the mahī". śrāvastī is a mistake for sarabhū here, but one can see how the sanskrit terms in the output might influence the n-gram based bleu and chrf++ scores, which compare it to the human reference translation that has similar terms. t2062 is an early 17th-century biography of a chinese monk. next to t1970 (a 12th century pure land treatise) it is the text in our sample for which the linguistic markers of "buddhist chinese" are least evident. it is thus not that surprising that our domainspecific model does not produce significant differences to gpt4 for those two texts. the language of t2062 differs from that of the other evaluation texts in that it contains many named entities, esp. person and place name, which are often referenced in an abbreviated way. the syntax is exceedingly terse with almost no redundancy or repetition. overall all models performed worst on t2062, with many passages translated wrongly to a degree that post-editing means retranslation. the example in the appendix is atypical in that it compares a relatively "easy" passage, which all models have managed to render reasonably well. as we have seen with t0026, wmt21+aug-ft tries to identify sanskrit terms and render them as such (jambudvīpa), but in this case unsuccessfully (ch. 荼 毗, skr. kṣapita). an interesting passage that shows how the context understanding of chatgpt3.5 is inferior to gpt4 and wmt21+aug-ft is 所聞種種，隨力不同 ("[all people] smelled something different, according to their powers [of insight]"). although in itself its choices are reasonable, chatgpt3.5 misses the subject with "various sounds and scents were heard, depending on the strength of the fire." note how the ambivalence of 聞 throws the model off. 聞 can indeed mean "to hear" or "to smell", but not both at the same time, in english "sounds and scents were heard" is nonsense. compared with the two large proprietary gptx models, the domain-specific wmt21+aug-ft model shows at least approximately equal, and often better, bleu and chrf++ scores. it needs to be remembered that inference on commercial gptx models costs orders of magnitude more than the finetuned wmt21 model, which can be efficiently served even on a single consumer-grade gpu @xcite . another problem when interacting with commercial models is the fact that their performance has been shown to differ significantly even in a relatively short amount of time, making their behavior unpredictable @xcite . ## conclusion for this paper we compiled a novel dataset for the training and evaluation of mt models for buddhist classical chinese. we applied two methods of data augmentation and compared a number of different encoder-decoder models finetuned on this data against large commercial mt providers and commercial decoder-only chat models. the domain-specific evaluation as well as the bleu/chrf++ scores show that with the help of the augmentation strategies, our much smaller and locally run model wmt21+aug-ft clearly outperforms the standard commercial providers as well as the commercial chat system chatgpt3.5, while being on par with gpt4 and outperforming it on certain domains. significantly, in the case of texts in the chinese canon that are originally translated from indic sources, our finetuned model outperforms gpt4 and is therefore the currently best available solution. for chinese-chinese buddhist texts, the performance of gpt4 is comparable to our models. this makes the finetuned model an ideal solution for semantic similarity tasks on corpus level, which are of central concern within the digital humanities and require cost-efficient and fast processing of large quantities of data. the evaluation results show that wmt21+aug-ft as well as gpt4 have reached a level of maturity that for the first time in the history of the translation of buddhist chinese texts into western languages, these tools can be of genuine help to translators. we see a number of directions for further work: first, the amount of digitized english translations aligned with their buddhist chinese counterparts is still very limited. increasing the size of this dataset promises further improvements in performance especially when it comes to chinese-chinese material, for which there is less bitext in the current dataset. second, while we present two strategies for data augmentation in this paper that are clearly boosting the performance of encoder-decoder models, further refinements of these approaches, especially the prompting of commercial engines with the right prior data, promise further significant leaps in performance. third, while llama2 has not shown competitive performance in our evaluation, the preliminary results are encouraging. by utilizing more monolingual data during the finetuning stage, we might see significant performance increases for smaller, open decoder-only models as well. fourth, our limited manual examination of the output of the mt models has indicated that widely used evaluation methods such as bleu or chrf++ do not always align well with human judgment, an observation that was made in other recent studies with a focus on large language models on mt as well @xcite . we therefore see a clear need for a thorough examination of alternative evaluation methods in future studies. consciousness as nonexistent, like the external world. some grasp the various kinds of consciousness as being the same in substance but different in characteristics. some believe that there are no separate mental functions apart from the mind. chatgpt3.5: there are those who hold mistaken views regarding the theory of the only consciousness. some believe that external objects are not without consciousness, while others believe that internal consciousness is not the same as external objects. some believe that various consciousnesses have different natures, while others believe that there is no separate mind apart from the mind. gpt4: furthermore, there are those who are confused and mistaken about the principles of consciousness-only, some clinging to the idea that external objects exist as consciousness and are not non-existent, some clinging to the idea that internal consciousness exists as objects and are not non-existent, some clinging to the idea that all consciousnesses function with the same substance, and some clinging to the idea that there is no separate mental object apart from the mind. wmt21+aug-ft: again, there are those who are deluded and mistaken about the principle of consciousness only. they may hold that external objects, like consciousness, are neither nonexistent nor existent, or they may hold that internal consciousness, like objects, is neither existent nor existent, or they may hold that the functions of the various forms of consciousness are different but the substance is the same, or they may hold that apart from mind there are no separate mental activities. t0026: 是時，晃昱諸天始生天者，不諳世 間成敗，不見世間成敗，不知世間成敗，見 大火已，皆恐怖毛竪而作是念： 『火不來至此 耶？火不來至此耶？』前生諸天諳世間成敗， 見世間成敗，知世間成敗，見大火已，慰勞諸 天曰： 『莫得恐怖，火法齊彼，終不至此。 』 human reference: then the gods of radiance, who have just been born as gods and have never experienced, seen, or known the world' s [cyclic] creation and destruction, on seeing that great fire will all be terrified, with their hair standing on end, and will think: "the fire will not reach up to here, will it? the fire will not reach up to here, will it?" those who were born as gods at an earlier time and have experienced, seen, and known the world's [cyclic] creation and destruction will, on seeing that great fire, calm the distress of those gods, telling them, "don't be afraid! that fire is of such a nature that it will not reach up to here." gpt-3.5: at that time, the heavenly beings who were born in the beginning of the world were ignorant of the success or failure of the world. they did not see the success or failure of the world, nor did they know the success or failure of the world. when they saw the great fire, they were all terrified and had the following thought: "has the fire reached here? has the fire reached here?" the heavenly beings from previous lives who were knowledgeable about the success or failure of the world, who saw the success or failure of the world, and who knew the success or failure of the world, comforted the other heavenly beings, saying: "do not be afraid. the fire will not reach here. it will be extinguished before it reaches here." gpt-4: at that time, the newly born gods, who were not familiar with the success and failure of the world, who did not see the success and failure of the world, and who did not know the success and failure of the world, saw the great fire and were all terrified, their hairs standing on end, and thought: "will the fire come here? will the fire come here?" the gods who had been born earlier, who were familiar with the success and failure of the world, who saw the success and failure of the world, and who knew the success and failure of the world, saw the great fire and comforted the other gods, saying: "do not be afraid, the law of fire is equal to that, it will never come here." wmt21+aug-ft: at that time, the ābhāsvara gods who had just been reborn among the gods did not study the formation and decay of the world, did not see the formation and decay of the world, and did not know the formation and decay of the world. but when they saw the great fire, they were all terrified, their hair standing on end, and they thought, 'will the fire not come here? will the fire not come here? 'the gods who had been reborn previously studied the formation and decay of the world, saw the formation and decay of the world, and knew the formation and decay of the world. but when they saw the great fire, they comforted the gods, saying, 'do not be afraid! the nature of fire is equal to that. it will never come here.' t2062 zhenqing: 荼毗日天色霽明，淨無纖 翳，舉火之際，忽有片雲，如蓋凝覆其上，灑 微雨數點。烟焰起時，異香充塞，內自殿閣僧 房，外自路人船子，所聞種種，隨力不同。火 餘骨有三色，而鏘鏘有聲。 human reference translation: on the day of his cremation, the weather had cleared up, and not a cloud was in the sky. but the moment the fire was lighted, a little cloud appeared suddenly. it grew denser just above the cremation site, like a canopy, and gently sprinkled a few drops of rain. when the fire arose, an extraordinary scent filled the air, and everybody, those in the monastery halls and monks'quarters, as well as those outside, the people on the roads and in their boats, all smelled something different, according to their powers [of insight]. the bones remaining in the ashes were of three colors and made a clear sound [when tapped]. chatgpt3.5: on the day of cremation, the sky cleared up, pure and without any blemish. just as the fire was being lit, suddenly a patch of clouds appeared, covering and sprinkling a few drops of light rain. as the smoke and flames rose, a strange fragrance filled the air, coming from the inner halls and monk's quarters as well as from passersby and boats on the road. various sounds and scents were heard, depending on the strength of the fire. after the fire subsided, the bones were left with three different colors and made a clanging sound. gpt4: on the day of the cremation, the sky was clear and bright, without a trace of cloud. at the moment the fire was lit, a patch of cloud suddenly appeared, covering the area like a canopy, sprinkling a few drops of light rain. when the smoke and flames rose, a strange fragrance filled the air, permeating from the inner temple halls and monks' quarters to the outer passersby and boats. various reactions were heard, differing according to one's perception. the remaining bones after the fire had three colors, and they made a clinking sound. wmt21+aug-ft: the day of jambudvīpa the sky was clear and clean, without the slightest cloudiness. while the fire was being raised, a cloud suddenly appeared, like a canopy, covering the fire and sprinkling a few drops of rain. when the smoke and flames arose, they were filled with an extraordinary fragrance. from inside the palace and the monks' quarters, and from outside the boats of the passers-by, the fragrance varied according to one's ability to smell it. the embers left by the fire were of three colors, and they emitted a sound when burned.