Large Language Models for P ersian- E nglish Idiom Translation (2025)

Abstract: Large language models (LLMs) have shown superior capabilities in translating figurative language compared to neural machine translation (NMT) systems. However, the impact of different prompting methods and LLM-NMT combinations on idiom translation has yet to be thoroughly investigated. This paper introduces two parallel datasets of sentences containing idiomatic expressions for Persian \rightarrow English and English \rightarrow Persian translations, with Persian idioms sampled from our PersianIdioms resource, a collection of 2,200 idioms and their meanings, with 700 including usage examples.Using these datasets, we evaluate various open- and closed-source LLMs, NMT models, and their combinations. Translation quality is assessed through idiom translation accuracy and fluency. We also find that automatic evaluation methods like LLM-as-a-judge, BLEU, and BERTScore are effective for comparing different aspects of model performance. Our experiments reveal that Claude-3.5-Sonnet delivers outstanding results in both translation directions. For English \rightarrow Persian, combining weaker LLMs with Google Translate improves results, while Persian \rightarrow English translations benefit from single prompts for simpler models and complex prompts for advanced ones.

## introduction an idiom is a phrase or expression with a figura tive meaning distinct from its literal interpretation. idioms are commonly used in everyday language to convey ideas more vividly and often originate from cultural, historical, or social contexts, making them specific to particular languages or regions. id iomatic expressions present significant challenges for nlp, particularly in translating between cultur ally distinct languages such as persian and english. despite their prevalence in spoken language, stateoftheart machine translation (mt) models struggle with translating idioms, often rendering them literally as compositional expressions (rau nak et al., @xcite . early mt efforts attempted to address this problem using id iom dictionaries or direct substitution @xcite . however, idioms evolve and vary by context, and even identical meanings can re quire different translations. for example, the idiom "keep at bay" has a different contextual meaning in the following sentences: (i) "the infection is kept at bay." meaning: "the infection is under control." (ii) "the fire keeps the wolves at bay." meaning: "the fire keeps the wolves away." recent large language models (llms) have en abled improvements in idiom translation compared to nmt models @xcite , due to their higher tendency towards nonliteralness @xcite and greater paraphrastic capability @xcite . however, no research has yet compared the performance of llms across dif ferent prompts, prompting techniques, and in com bination with nmt models. in this research, focusing on persianenglish translation, we try to fill several gaps. since persian datasets are limited in this context, we first introduce a comprehensive resource for id iomatic expressions in persian (persianidioms). this resource captures idiomatic expressions and their meanings, including contextual usage exam ples. additionally, we produce two parallel @xmath0, each consisting of 200 sen tences containing english and persian idiomatic expressions. we then use these datasets to eval uate the performance of opensource-qwen2.5 72b @xcite and command r+ (cohere for ai, 2024)-, and closedsource-gpt3.5 (ope nai, 2023), gpt4omini @xcite and claude3.5sonnet @xcite -llms, opensource-nllb2003.3b @xcite , madlad40010b @xcite -and closedsource-google translate-nmt models, and a combination of them in idiom translation across various settings. we manually assess translation quality using two metrics, idiom translation and fluency. then, we explore suitable replacements for manual evalua tion by calculating the correlation between scores from existing automatic evaluation approaches and manually obtained scores. in summary, our main contributions are as fol lows. (i) persianidioms, a new resource for persian idioms which includes about 2,200 idioms and their meanings-a resource that previously did not exist for persian. a subset of 700 idioms also includes example usages. (ii) parallel @xmath1. the persian sentences are sourced from persianidioms, while the english sentences are pri marily drawn from epie @xcite and magpie @xcite . (iii) a compre hensive comparison of human evaluations versus llmsasajudge, and existing automatic evalu ation methods in assessing translation quality of sentences containing idiomatic expressions. (iv) an evaluation of different prompting methods and a combination of llms and nmt models on idiom translation, highlighting their performance differ ences between @xmath2. these works demonstrate techniques for com piling figurative language data across multiple lan guages. however, they are focused on english or nonpersian languages, leaving a gap for a large scale persian idiom dataset. this research applies similar techniques of utilizing existing resources and language model generation to create idiom data specifically for persian. @xcite demonstrate that chatgpt com petes well with translation services like google translate for highresource european languages but struggles with lowresource or distant languages. @xcite evaluate the performance of gpt3.5 and gpt4' in adaptive mt, comparing it to strong mt systems and show that gpt3.5 ex cels in highresource languages but struggles with lowresource ones, where traditional models per form better. @xcite suggest that the increased tendency for paraphrasing in gpt trans lations could assist nmt models in translating fig urative language. we validate this hypothesis em pirically in our paper in the case of english and persian translations. yamada (2024) offers two prompts aimed at enhancing the quality of transla tions generated by chatgpt. we assess and con trast these prompts with our approaches. @xcite propose novel evaluation metrics for measuring translation literalness and compare the performance of gpt models and nmt models in id iom translation, finding that translations produced by gpt models are generally less literal. several other studies have been dedicated to comparing the performance of different llms and nmt mod els for different languages @xcite . others have aimed to enhance llm idiom translation through finetuning @xcite and augmenting them with knowledge bases @xcite . ## conclusion we introduced two parallel datasets for @xmath3. the persian id ioms were sampled from our persianidioms re source, with 2,200 idioms and their meanings. us ing these datasets, we evaluated multiple llms, nmt models, and their combination, focusing on idiom translation accuracy and fluency. our results show that claude3.5sonnet performs best in both directions. we also found that models generally translate english idioms more effectively than per sian ones. performance varies by translation direc tion-for @xmath4, combining weaker models with google translate improves their performance, and for @xmath5, weaker models performed best with single prompts. stronger models performed best with cot or multiple prompts in both directions. additionally, we evaluated existing automatic met rics and gpt4o as a judge, confirming gpt4o's reliability for assessing idiom translation accuracy.