Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs (2024)

Abstract: Translating text that contains entity names is a challenging task, as cultural-related references can vary significantly across languages. These variations may also be caused by transcreation, an adaptation process that entails more than transliteration and word-for-word translation. In this paper, we address the problem of cross-cultural translation on two fronts: (i) we introduce XC-Translate, the first large-scale, manually-created benchmark for machine translation that focuses on text that contains potentially culturally-nuanced entity names, and (ii) we propose KG-MT, a novel end-to-end method to integrate information from a multilingual knowledge graph into a neural machine translation model by leveraging a dense retrieval mechanism. Our experiments and analyses show that current machine translation systems and large language models still struggle to translate texts containing entity names, whereas KG-MT outperforms state-of-the-art approaches by a large margin, obtaining a 129% and 62% relative improvement compared to NLLB-200 and GPT-4, respectively.

## introduction the emergence of multilingual large language models (llms) and the wide availability of massive multilingual datasets have significantly advanced the field of machine translation (mt). these developments have led to mt systems that not only perform exceptionally well in high-resource languages but also support a growing number of lowresource languages (fan et al., 2021; tang et al., 2021; costa-jussà et al., 2022; kudugunta et al., * work carried out partially while at apple. these authors contributed equally. ## enhancing machine translation using multilingual knowledge graphs in contrast with augmentation strategies based on synthetic data that aim to maximize entity coverage at model training time, our hypothesis is that mt systems do not need to memorize every possible entity name transliteration and transcreation for each source-target language pair to correctly translate a text that contains entities. instead, the core idea that motivates our work is to leverage an external knowledge source to first retrieve the most relevant entities for an input text, and then generate the translation by incorporating the retrieved entity names in the target language. in fact, multilingual knowledge graphs, such as dbpedia @xcite , babelnet @xcite , and wikidata @xcite , provide a wealth of lexical and factual knowledge about millions of entities in many languages, including their names, aliases, and descriptions @xcite . not only that, but such knowledge is also easier to edit and is frequently updated to reflect the latest changes in the real world. by leveraging a multilingual knowledge graph, the focus of our approach shifts from memorizing entity names to learning when and how to retrieve the most relevant entities for a given input text and integrate their names in the target language in an end-to-end fashion. our method, kg-mt, features two main components: (i) a knowledge retriever, which retrieves the most relevant entities about the source text from a knowledge graph (section 3.1), and (ii) a knowledge-enhanced translator that generates the target text by incorporating the retrieved entity names (section 3.2). to better model the interactions between the retrieved entities and the translation, we also introduce a method to fuse the representations of the two components (section 3.3). figure 1 provides an overview of kg-mt, which we describe in detail in the following sections. ## evaluating cross-cultural translation of texts containing entity names to evaluate the effectiveness of our method, we introduce cross-culture translate (xc-translate), the first large-scale, manually-curated benchmark for the task of cross-cultural translation of texts containing entity names. xc-translate is composed of parallel texts in 10 english-to-x language pairs from a diverse set of languages, including both high-resource and low-resource languages, namely, arabic, chinese, french, german, italian, japanese, korean, spanish, thai, and turkish. we highlight that this design choice allows our benchmark to feature languages with diverse scripts, some of which are similar to english, such as french and spanish, and others that are very different, such as arabic, chinese, and thai. importantly, our benchmark is: we believe that xc-translate will be a valuable resource for the mt research community and will encourage further research on the problem of cross-cultural translation of texts containing entity names. ## experiments and results in this section, we first list the systems we consider in our main experiments, then describe the datasets used to train the mt systems, and finally report and discuss the results. systems. we compare the following systems: to ensure a fair comparison, we fine-tune mbart-50, m2m-100, nllb-200, and kg-mt using the same dataset. for more details about our experimental setup, please refer to the appendix. ## analysis and discussion in this section, we analyze kg-mt, and discuss where we believe it may be improved in future work. we expand on this analysis in the appendix. explicit or implicit knowledge? sections 3.2 and 3.3 introduce two methods to integrate the knowledge retrieved by the knowledge retriever into the knowledge-enhanced translator: explicit knowledge retrieval. the knowledge retriever plays a fundamental role in kg-mt. if the retrieval step collects wrong or unrelated entities, the knowledge-enhanced translator has to (i) be robust against noisy or irrelevant knowledge, and (ii) fall back on its parametric memory, which is often unreliable as shown by the results of the vanilla mt systems on xc-translate in tables 1 and 2 . however, our analysis shows that our knowledge retriever achieves 85.9% and 92.1% hits@1 and hits@3, respectively, on xc-translate, which suggests that the knowledge retriever is effective at retrieving relevant entities. part of this success can be attributed to our fine-tuning strategy with hard negative mining, which allows the knowledge retriever to improve the hits@1 and hits@3 by 5.6% and 4.2%, respectively, compared to using mcontriever @xcite , a pretrained retriever. given the good performance with hits@3, we set the knowledge retriever to retrieve at most three entities for each source text, which is a trade-off between retrieving more relevant entities and increasing the computational cost. gold knowledge. if knowledge retrieval is not a significant source of errors for kg-mt, then the knowledge integration step, in which the knowledge-enhanced translator has to learn how to effectively integrate the retrieved entities into the translation process, is likely to be the main bottleneck. to isolate the performance of the knowledgeenhanced translator from the performance of the knowledge retriever, we evaluate kg-mt when us- bleu 51.8 46.3 47.5 39.5 nllb kg-mt kg-mt kg-mt explicit implicit explicit or implicit knowledge integration? comet 84.6 83.9 83.1 81.9 meta 41.1 35.9 37.1 17.9 full bleu 56.3 51.8 39.5 nllb kg-mt kg-mt translation upper bound with gold knowledge comet 85.5 84.6 81.9 meta 52.7 41.1 17.9 gold knowledge ing gold entities instead of the entities retrieved by the knowledge retriever. figure 3 shows that the performance of kg-mt increases only by 11.6% in terms of average m-eta when using gold entities, indicating that the knowledge-enhanced translator is not always capable of using the gold knowledge. therefore, we hypothesize that the primary area of gain for future work shall be on improving the knowledge integration step, e.g., by using more sophisticated fusion strategies for knowledge integration or creating better datasets for fine-tuning the mt model on the knowledge integration step. ## conclusion and future work in this paper, we addressed the problem of crosscultural translation of texts containing entity names. our contributions are threefold: (i) we introduced kg-mt, a novel approach for retrieval-augmented translation that leverages the information available in multilingual knowledge graphs to improve the translation of texts containing entity names; (ii) we introduced xc-translate, the first large-scale, manually-curated benchmark for the task of crosscultural translation of texts containing entity names; and (iii) we conducted extensive experiments on xc-translate and other existing benchmarks for mt, showing that kg-mt significantly outperforms the state of the art on xc-translate while maintaining comparable results on general-purpose mt benchmarks. we believe that our contribution will encourage further research on the problems that arise when translating texts containing cultural references beyond entity names, such as idioms and metaphors, and that retrieval-augmented translation will be a valuable tool to address these challenges.