Prompting Large Language Models for Idiomatic Translation (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive performance in translating content across different languages and genres. Yet, their potential in the creative aspects of machine translation has not been fully explored. In this paper, we seek to identify the strengths and weaknesses inherent in different LLMs when applied to one of the most prominent features of creative works: the translation of idiomatic expressions. We present an overview of their performance in the EN \rightarrow IT language pair, a context characterized by an evident lack of bilingual data tailored for idiomatic translation. Lastly, we investigate the impact of prompt design on the quality of machine translation, drawing on recent findings which indicate a substantial variation in the performance of LLMs depending on the prompts utilized.

## introduction recent advancements in the field of artificial intelligence, particularly with the emergence of generative pre-trained transformer (gpt) models, have prompted the beginning of a new era of exploration into the applicability of large language models (llms) for machine translation tasks. the recent development and refinement of llms, such as gpt-3.5 and gpt-4 @xcite , have demonstrated their remarkable performance in understanding and generating natural language @xcite , thus positioning these models at the forefront of research into the translation of creative textual genres, including the nuanced task of translating idiomatic expressions. traditional neural machine translation (nmt) systems often falter in accurately capturing the essence of idiomatic expressions, tending towards translations that are either overly literal or misinterpret the intended meaning. in contrast, recent studies have illustrated the ability of gpt models to adopt less literal translation approaches, especially in handling idiomatic expressions, leveraging an enhanced understanding of context and figurative language. this contribution will evaluate various large language models (llms) to establish benchmarks for their effectiveness in translating idiomatic expressions in the english-italian language pair. the objective is to identify the strengths and weaknesses inherent in different llms when applied to machine translation (mt) tasks, particularly focusing on the nuanced aspect of creative language. furthermore, the study will explore the impact of prompt design on mt quality, drawing on the findings of @xcite that suggest that the performance of llms in multilingual tasks can vary significantly with the prompts used. through these evaluations, we seek to contribute to the improvement of machine translation technologies, highlighting the potential of llms to make creative works more accessible across languages, enriching cultural exchange and overcoming language barriers. may have on the model output @xcite @xcite . the second line focuses on the evaluations of llms in various translation scenarios, covering multilingual @xcite @xcite , document-level @xcite @xcite , low-resource translation @xcite , hallucination @xcite and domain adaptation @xcite . this study positions itself within the second research axis, concentrating on the evaluation of llms in specialized translation scenarios. despite the large body of research currently being conducted on llms performance, research to date has not yet fully explored their application to the translation of creative texts. this study does not aim to provide a comprehensive overview of the topic, but we seek to evaluate the intricate task of translating idiomatic expressions, a critical aspect that challenges the adaptability and understanding of these models. ## experimental setup in this section, we describe the methodology used in our experiments, including the translation process and the evaluation metrics employed. we initiated the translation process leveraging openai api and the huggingface library @xcite in python, generating four batches of translations using four distinct prompts applied to the models gpt-3.5-turbo and mistral-7b-v0.1. for the machine translation (mt) evaluation, we used the online evaluation platform mateo @xcite , which provides an easy-to-use user interface for the evaluation of translations, utilizing state-of-the-art neural and n-gram evaluation metrics. we conducted the experiment in three independent trials to ensure the reliability of the results and replicability of the experiment. ## evaluation we present a comprehensive evaluation of the four prompt templates we have selected, using two models: gpt-3.5-turbo-1106 and mistral-7b-v-0.1. our evaluation used two mainstream neural evaluation metrics: comet @xcite and bleurt @xcite . these metrics have shown a very high correlation with human judgment and are established in the evaluation of llm-based machine translation @xcite . we have decided to include bleu @xcite in our evaluation, as it remains a widely recognized standard metric in mt evaluation, despite its limitations for our specific translation context. more specifically, in the context of accurately conveying idiomatic expressions into another language, there is frequently a mismatch between the length of the sentence in the source and target texts. metrics such as bleu and chrf @xcite may not be the most adequate for the task, as they tend to penalize length, lexical discrepancies and brevity of the translations, which are not necessarily indicative of poor translation quality, especially in the context of idiomatic expressions. ## conclusions this work presents a preliminary analysis on the use of llms for the translation of idiomatic expressions. we find that, given the same dataset and task, identical prompts may have optimal efficacy across various models, as seen for mistral-7b and gpt-3.5, and that it is possible to optimize the model's performance by choosing an optimal prompt. in our experiments, the fiveshot prompt (prompt d) consistently outperformed other prompts in terms of bleurt and comet across both models, over three independent trails, demonstrating the efficacy of leveraging in-context learning ability to improve the model's understanding of idiomatic expressions. as for zeroshot prompting, prompt c consistently performed the best. we find that gpt-3.5 consistently outperforms mistral-7b which, on the other hand, can come close to gpt's performance when prompted correctly. finally, we underline the limitations of traditional metrics based on n-grams, such as bleu, in evaluating the translation of idiomatic expressions, advocating the use of neural-based evaluation metrics that better capture semantic nuances. overall, our findings promote a more strategic approach to prompt selection and model use in machine translation, pointing towards a future where llms can be used effectively for nuanced and culturally-specific translation tasks. as the field of mt continues to evolve, so too will the strategies for leveraging the full potential of large language models in understanding and translating the rich nuances of human language.