Creative and Context-Aware Translation of E ast A sian Idioms with GPT -4 (2024)

Abstract: As a type of figurative language, an East Asian idiom condenses rich cultural background into only a few characters. Translating such idioms is challenging for human translators, who often resort to choosing a context-aware translation from an existing list of candidates. However, compiling a dictionary of candidate translations demands much time and creativity even for expert translators. To alleviate such burden, we evaluate if GPT-4 can help generate high-quality translations. Based on automatic evaluations of faithfulness and creativity, we first identify Pareto-optimal prompting strategies that can outperform translation engines from Google and DeepL. Then, at a low cost, our context-aware translations can achieve far more high-quality translations per idiom than the human baseline. We open-source all code and data to facilitate further research.

## introduction figurative language is a challenge for both linguistic analysis @xcite and many natural language processing (nlp) tasks @xcite . one representative task is literary translation @xcite , where the translation of figurative language is one major difficulty. among figurative language constructs, idioms are especially hard for a machine translation (mt) model due to their non-compositionality. for example, the meaning of the idiom "bite the bullet", deciding to do something difficult, is not simply composed of the meanings of "bite" and "bullet". of idioms in all languages, east asian idioms constitute an interesting subset. each of these idioms condenses its figurative meaning into a small number of characters, dominantly 4 characters (chinese: sizichengyu, japanese: yojijukugo, 1. idiom 刮目相看 2. sentences 小明的成绩提高得非常快，让老师和同学们 都刮目相看。 ## method in this section, we elaborate on the methods and experiment details for each step shown in step 1: idioms we obtain idioms from a dictionary for chinese @xcite and online resources for japanese 2 and korean 3 . these sources cover commonly used idioms in the 3 east-asian languages. to test if our method generalizes to uncommon or new words that have an idiom-like structure, we also curate a set of plausible chinese idioms. plausible idioms are gpt-4-generated words which are not real idioms, but can fool gpt-4 when we ask it if the word is an idiom (appendix a.1). for convenience, we use "plausible chinese" to refer to the language of these idioms for convenience. for the 4 source languages, we limit the target language to english. to our best knowledge, the only human baseline that provides multiple translations for each idiom is the chinese-english dictionary we use. ## results and discussion in this section, we first analyze the generated translations. then, we examine how the optimal translation strategies work on a larger set of idioms. we also discuss how the dataset can be extended by an alternative pipeline using paragraph context. ## related work different from traditional mt models, llms can produce diverse and less literal translations @xcite . hence, a series of work has targeted generating diverse translations with llms and selecting the best. on one hand, translations can be generated from scratch using various prompting strategies. then, ensembling methods can be applied to select the best candidates @xcite . on the other hand, candidates can be further refined. some examples include refining candidates generated by other machine translation systems @xcite , iterative editing @xcite , self-correction @xcite , and multi-agent debate @xcite . while existing work focused mostly on general translation, our work contributes in the more challenging task of generating diverse, high-quality translations for idioms. ## conclusion in this work, we thoroughly test prompting strategies that generate different translations for an east asian idiom. we identify the strategies that generate most creative and faithful translations. to our surprise, the prompts derived from human experience do not consistently generate quantitatively better translations. finally, we use the pareto optimal strategies to construct a dataset of high quality translations, which can help human translators.