Chain-of-Dictionary Prompting Elicits Translation in Large Language Models (2024)

Abstract: Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation (MNMT) even if not being trained explicitly for translation. Yet, they still struggle with translating low-resource languages. As supported by our experiments, a bilingual dictionary between the source and the target language could help. Motivated by the fact that multilingual training effectively improves cross-lingual performance, we show that a chained multilingual dictionary with words expressed in more languages can provide more information to better enhance the LLM translation. To this end, we present a novel framework, CoD, Chain-of-Dictionary Prompting, which augments LLMs with prior knowledge with the chains of multilingual dictionaries for a subset of input words to elicit translation abilities for LLMs. Experiments indicate that ChatGPT and InstructGPT still have room for improvement in translating many language pairs. And CoD elicits large gains by up to 13x chrF++ points for MNMT (3.08 to 42.63 for English to Serbian written in Cyrillic script) on FLORES-200 full devtest set. We demonstrate the importance of chaining the multilingual dictionaries, as well as the superiority of CoD to few-shot in-context learning for low-resource languages. Using CoD helps ChatGPT to obviously surpass the SOTA translator NLLB 3.3B.

## introduction large language models (llms) possess the ability to carry out high-quality machine translation tasks without specific training, as observed in previous studies @xcite @xcite @xcite . llms can be prompted to do so by requesting them to complete a prompt, such as "translate the following sentence to english from french:" followed by an input sentence written in french. however, despite their training on extensive datasets, these models may encounter difficulties in correctly translating rare words that frequently occur in low-resource situations. motivated by such a lexical-level problem, we seek how to incorporate dictionaries for improving mnmt. further, motivated by the fact that multilingual training effectively improves cross-lingual performance @xcite @xcite , we use multilingual dictionaries to enhance the translation performance of llm prompting. to this end, we leverage the multilingual dictionaries as the prior knowledge, and we describe a method to prompt llms with hints that indicate a set of possible chained multilingual translations for specific words in the input. this method involves adding a string such as "'limit' means 'grenze' means 'çäk'." to the start of the standard machine translation prompt as lexicon hints for mt. this approach is motivated by the fact that supervised machine translation models have effectively used dictionaries to enhance translation @xcite @xcite . we also propose the method as a chain of dictionary in the light of chain-of-thought (cot) reasoning @xcite that represents the reasoning procedure as intermediate thinking steps. in our case, we show how to incorporate multilingual knowledge in a zero-shot manner by chaining the translations of words across various languages to improve llm's mnmt capabilities. this allows us to specify the task in the prompt and provide background knowledge that is useful in completing the task of machine translation, without placing any strict constraints on how the model employs this knowledge, as demonstrated in figure 1 . we conducted extensive experiments with the translate the following text from english into tamil: ## chain-of-dictionary prompting for neural machine translation large language models show their promising translation performance when sufficiently pre-trained @xcite . however, this is frequently not the case, especially for these lowresource languages. there are thousands of languages around the world, and current research on mt has scaled to at least 200 @xcite . it is an important research topic to explore the capabilities of llms to cover as many languages as possible. despite the importance of covering lowresource languages in llms, we will report in this paper that the latest llms are still far from satisfying in covering these low-resource languages from @xcite . we propose a novel framework called cod (chain-of-dictionary prompting) to address these difficulties by chaining multilingual dictionary knowledge into prompting-based machine translation. compared to in-context learning that uses few-shot demonstrations to prompt the llms, dictionaries are comparatively easier to store and acquire than the demonstrations, particularly for low-resource languages @xcite @xcite . this makes cod an attractive external resource for mt with llms. our novel approach, cod, utilizes promptingbased translation and integrates chained multilingual dictionary information as prior knowledge directly into the prompt. when presented with a source sentence, we search for the multilingual dictionary entries for a subset of the words: before making the conventional translation request to llms, we append additional textual inputs to the prompt that outline possible chained multilingual translations for those specific words. therefore, the prompts for each sentence consist of two parts, as illustrated in figure 1: (1) the translation prompt: "translate the following text @xmath0: <source-@xmath1. we do not include few-shot in-context learning in our methodology as we inspected that it is usually hard to retrieve relevant demonstrations for low-resource languages, which yields limited improvements. in the remaining sections, we will report relevant experimental results which indicate that few-shot demonstrations are less favourable to our methods for low-resource translations. we also found that using non-chained decomposed multilingual dictionaries instead of cod degrades the results: ## conclusions cod is a novel framework that uses chained multilingual dictionaries when prompting large language models (llms) for mnmt. we evaluate chatgpt, instructgpt, and bloom on the flores-200 dataset for mnmt. we found that chatgpt and instructgpt still have room for improvement in translating many language pairs. cod elicits large gains by up to 13x chrf++ points for mnmt (3.08 to 42.63 for english to serbian written in cyrillic script) on flores-200 full devtest set. we also verified the necessity of the chained multilingual dictionaries, and we found that both of them are quite important to cod. cod also outperforms few-shot demonstrations which struggle to retrieve relevant demonstrations for low-resource settings. cod can even surpass the strong sota nllb translator in translation. extensive case studies demonstrate that cod elicits translation even if the words are not directly presented by cod. there are over 7,000 languages around the world, and cod is the first work that scales the translation capability of llms to over 200 languages. we hope that cod can help researchers to improve cross-lingual performance on neural models further.