Retrieval-Augmented Machine Translation with Unstructured Knowledge (2025)

Abstract: Retrieval-augmented generation (RAG) introduces additional information to enhance large language models (LLMs). In machine translation (MT), previous work typically retrieves in-context examples from paired MT corpora, or domain-specific knowledge from knowledge graphs, to enhance MT models. However, a large amount of world knowledge is organized in unstructured documents, and might not be fully paired across different languages. In this paper, we study retrieval-augmented MT using unstructured documents. Specifically, we build RAGtrans, the first benchmark to train and evaluate LLMs’ retrieval-augmented MT ability. RAGtrans contains 169K MT samples collected via GPT-4o and human translators. Besides, documents from various languages are also provided to supply the knowledge to these samples. Based on RAGtrans, we further propose a multi-task training method to teach LLMs how to use information from multilingual documents during their translation. The method uses existing multilingual corpora to create auxiliary training objectives without additional labeling requirements. Extensive experiments show that the method improves LLMs by 1.6-3.1 BLEU and 1.0-2.0 COMET scores in En-Zh, and 1.7-2.9 BLEU and 2.1-2.7 COMET scores in En-De. We also conclude the critical difficulties that current LLMs face with this task.

## introduction retrieval-augmented generation (rag) has grown into a practical paradigm in the development of large language models (llms). with the help of retrieved information, llms could generate more accurate and knowledge-enrich responses @xcite . previous work brought rag into machine translation (mt), and could be mainly classified into the following two streams: (1) retrieving in-context examples (also known as "translation memory"): for a source sentence, a few studies retrieve the relevant paired sentences from bilingual corpora to enhance mt models @xcite @xcite @xcite . further, @xcite relax the bilingualism limitation, and try to directly retrieve similar target-language translations to enhance models. (2) retrieving knowledge triplets: the others retrieve relevant information from knowledge graphs to let the models know domain or cultural knowledge w.r.t. the source sentences @xcite . despite the great success that has been achieved, a large amount of world knowledge is organized in unstructured documents, and might not be fully paired across different languages. this unstructured knowledge is neglected by previous work. for example, wikipedia serves as an encyclopedia of world knowledge. most of its information is listed in documents. besides, for a piece of specific knowledge, wikipedia does not always provide it in all languages. though multilingual information of some general knowledge is provided, their content might be differentiated among different languages @xcite . in this paper, we study retrieval-augmented mt using unstructured documents. since we are the first to study this topic and previous datasets do not support the research, we first build a benchmark dataset, named ragtrans. in detail, ragtrans is collected based on wikipedia with three key features: (i) knowledge-intensive sentences: ragtrans randomly selects 169k english sentences from wikipedia as the source sentences, which generally come from the lead paragraphs of different wikipedia pages, containing knowledge-intensive semantics. thus, understanding these source sentences tends to require additional knowledge. (ii) useful relevant documents: to achieve retrieval-augmented mt, for each source sentence, its following content on the wikipedia page (in english) could serve as its relevant document. (iii) transferability to multilingual rag: wikipedia also provides multilingual parallel content. therefore, for a source sentence, its relevant knowledge in different languages can also serve as relevant documents. as a result, mt models can leverage knowledge from multilingual documents beyond the source and the target languages. in this work, we choose chinese, german, french and czech. after collecting the source english sentences and relevant documents, we randomly split them into training, validation and testing sets. for training and validation samples, we employ @xcite to collect the chinese or german translation; while we employ professional human translators to perform the same process for the testing samples. finally, ragtrans involves 79k retrieval-augment en⇒zh and 90k en⇒de mt samples foot_0 , each of which contains an english source sentence, a document in english, chinese, german, french or czech, and the corresponding translation. based on ragtrans, we train llms and evaluate their retrieval-augmented mt performance from the following settings: (1) golden evaluation: providing llms with the golden relevant documents during data collection, and testing the translation performance. (2) robustness evaluation: providing irrelevant documents to test the llms' robustness. furthermore, during the application phase of a retrieval-augmented mt model, the model might receive multiple documents from various languages. these multilingual documents are not restricted to parallel documents and can convey diverse meanings. in light of this, we propose a multitask training method to enhance llms' ability to leverage multilingual knowledge. specifically, we design three training objectives, i.e., cross-lingual information completion, self-knowledge-enhanced translation and cross-lingual relevance discrimination. among them, cross-lingual information completion and cross-lingual relevance discrimination train llms to refine and judge information from multilingual documents. self-knowledge-enhanced translation lets llms generate relevant knowledge in various languages for the source sentences, and then perform mt with the help of its multilingual self-knowledge. the multi-task training samples of these objectives can be automatically created from existing multilingual corpora, and do not need any additional labeling costs. experiments on ragtrans show that the multi-task training method improves llms' ability to leverage relevant knowledge. using qwen2.5-7b @xcite as the backbone, compared with simply instructiontuning on ragtrans, the retrieval-augmented mt performance is improved by 1.6∼3.1 bleu and 1.0∼2.0 comet scores in en⇒zh, and 1.7∼2.9 bleu and 2.1∼2.7 comet scores in en⇒de. finally, we discuss specific challenges that current approaches faced with this task and give multiple promising directions for future research. our main contributions are concluded as follows: • to the best of our knowledge, we are the first to study retrieval-augmented mt using unstructured knowledge. to this end, we construct the first corresponding benchmark dataset, i.e., ragtrans, containing 169k translation samples collected via gpt-4o and human translators. ## ragtrans in this section, we first discuss how we select english source sentences and their relevant documents from wikipedia ( § 2.1). then, we introduce the details of the data translation via gpt-4o and human translators ( § 2.2). finally, we give statistical analyses of ragtrans ( § 2.3), and provide the details of benchmark settings ( § 2.4). ## multi-task training to further enhance llms' retrieval-augmented mt ability, we propose a multi-task training method, named csc, which involves three designed train-ing objectives, i.e., cross-lingual information completion, self-knowledge-enhanced translation and cross-lingual relevance discrimination. in this section, we first introduce these objectives ( § 3.1) and then discuss how to create their training samples from existing corpora ( § 3.2). ## conclusion in this paper, we explore the retrieval-augmented mt with unstructured knowledge. to this end, we build ragtrans dataset with 169k retrievalaugment mt samples to train and evaluate llms' retrieval-augmented mt ability. further, we propose csc multi-task training method with three designed objectives to teach llms to leverage multilingual knowledge in retrieval-augmented mt. extensive experiments demonstrate the usability of