What is the Best Way for C hat GPT to Translate Poetry? (2024)

Abstract: Machine translation (MT) has historically faced significant challenges when applied to literary works, particularly in the domain of poetry translation. The advent of Large Language Models such as ChatGPT holds potential for innovation in this field. This study examines ChatGPT’s capabilities in English-Chinese poetry translation tasks, utilizing targeted prompts and small sample scenarios to ascertain optimal performance. Despite promising outcomes, our analysis reveals persistent issues in the translations generated by ChatGPT that warrant attention. To address these shortcomings, we propose an Explanation-Assisted Poetry Machine Translation (EAPMT) method, which leverages monolingual poetry explanation as a guiding information for the translation process. Furthermore, we refine existing evaluation criteria to better suit the nuances of modern poetry translation. We engaged a panel of professional poets for assessments, complemented evaluations by using GPT-4. The results from both human and machine evaluations demonstrate that our EAPMT method outperforms traditional translation methods of ChatGPT and the existing online systems. This paper validates the efficacy of our method and contributes a novel perspective to machine-assisted literary translation.

## introduction foreign poems translated into chinese still have to be like poems. for chinese poetry, there are significant differences between various genres. classical chinese poetry is characterized by strict constraints on format, meter, sentence length, and rhyme. in contrast, modern poetry is free from these constraints and breaks away from the rigid structures of classical poetry @xcite . previous work has successfully applied machine translation to poetry, but these poems typically have clear format or rhyme restrictions @xcite @xcite @xcite , which differ significantly from modern chinese poetry. recent research has taken an innovative approach by first obtaining an initial translation of the input prose using traditional neural machine translation (nmt) methods. this initial translation is then mapped to a set of masked sequences via a designed heuristic method. finally, these sequences are used to generate poetry translations through a pre-trained masked language modeling (mlm) technique @xcite . another study compared the differences between machine translation and human translation of arabic poetry into english (alowedi and al-ahdal, 2023). the authors concluded that machine translation is not suitable for translating arabic poetry into english as it fails to comprehend the socio-cultural background of poetry creation and the contextual nuances, particularly the genre-specific elements. promisingly, the artificial intelligence chatbot chatgpt, released by openai, has demonstrated excellent performance across various tasks and domains, including translation tasks @xcite @xcite . although previous work has studied the performance of chatgpt on translation tasks @xcite @xcite , and recent studies have explored the application of chatgpt to poetry-related tasks, these investigations primarily focused on poetry generation. for instance, recent research examined the effectiveness of chatgpt-4 in generating arabic poetry and found the results to be unsatisfactory @xcite . the study highlighted several issues with the text generated by chatgpt-4, including poor language quality, superficial content, lack of emotion, inconsistent speech, inappropriate word choices, and an ease of recognition by human evaluators. unlike previous work, this paper focuses on the capabilities of chatgpt in translating english po-etry into modern chinese poetry. we explored optimal strategies for utilizing chatgpt to translate poetry and evaluated its maximum performance in this specific task. inspired by @xcite and @xcite , we investigated chatgpt's performance on modern poetry translation tasks by designing appropriate prompts and providing example shots for the model. experimental results demonstrate the effectiveness of our designed prompts. despite these promising outcomes, our analysis reveals persistent issues in the translations generated by chat-gpt that warrant further attention. consequently, we propose a new poetry translation method called explanation-assisted poetry machine translation (eapmt). our method leverages the explanation of monolingual poetry as guidance information to achieve high-quality translations from english poetry to modern chinese poetry. furthermore, existing evaluation criteria are typically designed for ordinary texts or poems with specific restrictions and are not fully applicable to modern poetry. therefore, we refined these criteria to better capture the nuances of contemporary poetry translation. we engaged a panel of professional poets for assessments and complemented their evaluations with those conducted using gpt-4. the results from both human and machine evaluations demonstrate that our eapmt method outperforms traditional translation techniques of chatgpt and existing online systems. the contributions of our work are as follows: ## related work the earliest work on poetry machine translation employed phrase-based machine translation systems to translate french poetry into metrical english poetry @xcite . this research explored how statistical machine translation systems could produce translations that adhere to constraints such as length, rhythm, and rhyme. subsequently, @xcite used statistical methods to translate rhymed poetry, achieving the translation of italian poetry into english. through unsupervised learning, they identified word stress patterns in an original poetry corpus, which were then utilized in generating english poetry. @xcite introduced the first neural poetry translation system, capable of translating french poetry into english while adhering to user-specified target rhythm and rhyme patterns. this system demonstrated greater flexibility than phrase-based machine translation (pbmt) systems, consistently generating translations that fit any specified scheme. @xcite conducted an empirical study on poetry translation, highlighting a crucial yet often overlooked issue: advanced machine translation systems, trained on large amounts of non-poetry data, struggle to preserve poetic style even if they can maintain meaning and fluency. in 2022, a hybrid machine translation (hbmt) model was proposed @xcite , which improves the semantic and syntactic accuracy of the translation framework. this model effectively translates hindi poetry into english, demonstrating notable advancements in maintaining the integrity of the original poetic content. ## construction of modepoem dataset translators need to have a strong foundation in both the source and target languages @xcite . additionally, poetry translators must understand poetry and ideally write poems themselves. in this paper, we constructed a high-quality bilingual poetry dataset, modepoem, consisting of 400 modern poems (7,000 lines) in english and their chinese translations. our dataset is selected from the high-quality online platform poetrysky. 2 we were meticulous in our selection process, ensuring that only high-quality data was included. this rigor is in part due to the expertise of one of our authors, a professional poet who has rigorously filtered out any data of inferior quality. it is important to note that the judges involved in our human evaluations did not overlap with the translators of the dataset, ensuring unbiased assessments. the source poems in this dataset were written by multiple professional english poets, with translations from various bilingual poetry translators who are native chinese speakers. these translators are also accomplished poets, proficient in english, and have extensive experience translating poetry from english to chinese. therefore, the translations in this dataset can be considered the gold standard. the statistics of our dataset are shown in ## explanation-assisted poetry machine translation in light of the experimental results and analysis presented, it is evident that while chatgpt can interpret and analyze poems (chen and chen, 2023), it still encounters numerous challenges. to enhance the performance of chatgpt in poetry translation, we introduce a novel approach termed explanation-assisted poetry machine translation (eapmt). this method leverages monolingual poetry explanations as guidance information. specifically, the process begins with chatgpt generating an explanation for a poem presented in the source language. this explanation encompasses both the literal content and deeper meanings, contextualized within the cultural backdrop of the poem. subsequently, this explanation serves as guidance for chatgpt to produce a translation into the target language. the two-step framework ensures that nuances and subtler aspects of the poem, often lost in conventional translation processes, are preserved. figure 1 contrasts the novel explanation-assisted poetry machine translation (eapmt) framework with traditional translation methods, highlighting the advantages of the eapmt approach. ## experiments our experiments were performed using gpt-3.5 (gpt-3.5-turbo) and gpt-4 (gpt-4-1106), and involved comparisons with leading online translation systems. for chinese literary texts, baidu translate foot_4 demonstrated superior performance over google translate @xcite . consequently, baidu translate was selected as the online translation system for our study. ## conclusion in this work, we examine chatgpt's capabilities in english-chinese poetry translation tasks by utilizing targeted prompts and small sample scenarios to ascertain optimal performance. to address chat-gpt's shortcomings, we propose an explanation-assisted poetry machine translation (eapmt) method, which leverages monolingual poetry explanation as a guiding information for the translation process. furthermore, we refine existing evaluation criteria to better suit the nuances of contemporary poetry translation. the results from both human and machine evaluations demonstrate the effectiveness of our eapmt method. in future work, we will try to correct the explanation generated in the first step with human experts or chatgpt, thereby further improving the accuracy of the translation.