Morphological and Semantic Evaluation of A ncient C hinese Machine Translation (2023)

Abstract: Machine translation (MT) of ancient Chinese texts presents unique challenges due to the complex grammatical structures, cultural nuances, and polysemy of the language. This paper focuses on evaluating the translation quality of different platforms for ancient Chinese texts using The Analects as a case study. The evaluation is conducted using the BLEU, LMS, and ESS metrics, and the platforms compared include three machine translation platforms (Baidu Translate, Bing Microsoft Translator, and DeepL), and one language generation model ChatGPT that can engage in translation endeavors. Results show that Baidu performs the best, surpassing the other platforms in all three metrics, while ChatGPT ranks second and demonstrates unique advantages. The translations generated by ChatGPT are deemed highly valuable as references. The study contributes to understanding the challenges of MT for ancient Chinese texts and provides insights for users and researchers in this field. It also highlights the importance of considering specific domain requirements when evaluating MT systems.

## introduction machine translation (mt) has been a prominent area of research and development in artificial intelligence since the 1950s. over the years, it has undergone significant advancements, evolving from rule-based methods, statistical methods, and more recently, neural network-based learning methods. as the quality of mt continues to improve and the demand for translation work steadily increases, more and more translators are adopting the "machine translation + post-editing" mode for translation. at the same time, the quality of mt has been a subject of great interest and concern for both the mt and translation fields. researchers, institutions, and conferences are continuously conducting studies in this area, and various evaluation metrics for mt have been proposed. there have also been studies on mt of ancient texts. some researchers have made algorithmic improvements specifically tailored for translating ancient texts @xcite @xcite . researchers have also conducted evaluations of the quality of mt for ancient texts @xcite @xcite . however, research on mt for ancient texts, including ancient chinese texts, remains relatively scarce. this paper primarily focuses on mt quality of ancient chinese texts, and the subsequent discussions will concentrate on this specific domain. compared to modern chinese, ancient chinese has its own unique characteristics. firstly, ancient chinese employs complex and distinctive grammatical structures, including syntax, word order, and rhetoric, among other aspects. these structures differ significantly from modern chinese. mt struggles to accurately capture and parse the intricate grammatical relationships embedded in ancient chinese texts. secondly, ancient chinese texts often employ rhetorical devices such as allusions, symbolism, and metaphors, which involve rich cultural connotations and backgrounds. these allusions and cultural nuances are often challenging for non-chinese mt systems to comprehend, leading to translation errors or the loss of the original essence and aesthetic appeal. thirdly, ancient chinese texts often exhibit polysemy and ambiguity, where a single word or phrase may have multiple interpretations and meanings. mt systems find it challenging to accurately select and judge among these complex semantic relationships, often leading to mistranslations or inaccuracies. the aforementioned characteristics pose significant challenges for mt of ancient chinese texts. this study aims to evaluate the translation quality of different platforms for ancient chinese texts. through this evaluation, we can gain insights and understanding in dealing with the complexities of ancient language and culture, contribute to the advancement in the field of natural language processing, and provide a supplement to mt quality assessment applications. furthermore, these evaluation results will help users gain insights into the performance of different platforms, allowing them to identify potential issues and limitations. ## experiment design this study takes the chinese classic the analects foot_0 as the research text and compares three classic human-translated versions and four versions generated by four platforms. three mt quality evaluation metrics are used as evaluation criteria to assess the translation quality of the four platforms. for each human-translated text and each machinetranslated text, quality scores are calculated individually. then, the mean scores are calculated for each platform. the scheme is illustrated in figure 1 . ## other findings: chatgpt's advantages from example 1 source text ("st" for short): 君子不器。 baidu: a gentleman is not a weapon. bing: a gentleman is not a weapon. deepl: a gentleman is not a tool. chatgpt: a noble person is not defined by their possessions or material objects. in this context, "器" refers to "utensil." this sentence conveys the idea that a noble person (君 子) should have a broad and inclusive mindset, rather than being limited to a specific role or function like a utensil (器具). however, baidu and bing mistakenly interpret "器" as "weapon" (武器) and deepl "tool" (工具). example 2 st: 诲女知之乎！ baidu: teach women to know! bing: the girl knows! deepl: teach your daughter to know! chatgpt: have you learned to understand this? in ancient chinese, the character "女" (female) is often used as a phonetic loan character, representing the pronoun "汝" (thou). in example 2, baidu, bing, and deepl do not correctly identify the true intended meaning of "女," while chatgpt do. we have found that, in our research, the translation of all instances of the character "女" as a phonetic loan character in the analects exhibits the same pattern across different mt translations. when the text with errors is input into the platforms, baidu, bing and deepl generate translations based on the faulty text. however, chatgpt often demonstrate the ability to recognize and identify these errors and generate translations based on the correct characters. chatgpt possesses this capability that is not present in traditional mt systems. example 3 st: 邦君之妻，君称之曰夫人，夫人自称日 小童…… baidu: bangjun's wife, whom you call madame, and whose wife calls herself rixiaotong… bing: the wife of the prince, the prince is called mrs. wang, and the lady calls herself ri xiaotong… deepl: the wife of the king of the state, the king called it the wife, the wife called herself the day child… chatgpt: the wife of the ruler of a state is called "madam" by the ruler, and she refers to herself as "little child"… in example 3, the underlined character "日" (pronounced as "ri", meaning the sun or day) in the source text should be "曰" (say), and "小童" that follows means "little child". this sentence means that the madam calls herself "小童. chatgpt: if one is respectful without propriety, it leads to laboriousness. if one is cautious without propriety, it leads to timidity… "蒽" (anthraquinone) in the original text is a faulty character which should be "葸" (afraid, bashful). both baidu and deepl take "蒽" directly, and bing simply omits it. chatgpt recognizes the fault and translates it into "timidity," which aligns with the intended meaning of the original text. ## conclusion based on the evaluation of translation quality of the chinese classic the analects by baidu, bing, deepl, and chatgpt using the bleu, lms, and ess metrics, we have found that among the four platforms, baidu, as a mt platform developed in china, performs the best in handling ancient chinese texts. its scores in all three metrics are significantly higher than the other three platforms. chatgpt, as a general-purpose language model, ranks second among the four, and has demonstrated unique advantages, and the translations it produces are highly valuable as references. it is worth mentioning that in this study, the translations generated by chatgpt were done without any prompts (except for the one mentioned in 2.2) or adjustments. we plan to discuss in our future research the translation quality of chatgpt by incorporating prompts for adjusting the translation of ancient chinese into english.