Leveraging Multilingual Knowledge Graph to Boost Domain-specific Entity Translation of C hat GPT (2023)

Abstract: Recently, ChatGPT has shown promising results for Machine Translation (MT) in general domains and is becoming a new paradigm for translation. In this paper, we focus on how to apply ChatGPT to domain-specific translation and propose to leverage Multilingual Knowledge Graph (MKG) to help ChatGPT improve the domain entity translation quality. To achieve this, we extract the bilingual entity pairs from MKG for the domain entities that are recognized from source sentences. We then introduce these pairs into translation prompts, instructing ChatGPT to use the correct translations of the domain entities. To evaluate the novel MKG method for ChatGPT, we conduct comparative experiments on three Chinese-English (zh-en) test datasets constructed from three specific domains, of which one domain is from biomedical science, and the other two are from the Information and Communications Technology (ICT) industry — Visible Light Communication (VLC) and wireless domains. Experimental results demonstrate that both the overall translation quality of ChatGPT (+6.21, +3.13 and +11.25 in BLEU scores) and the translation accuracy of domain entities (+43.2%, +30.2% and +37.9% absolute points) are significantly improved with MKG on the three test datasets.

## introduction recently, the emergence of chatgpt foot_0 has brought remarkable influence on natural language processing (nlp) tasks. it is an intelligent chatbot developed by openai, based on instruct-gpt @xcite . chatgpt is built on the top of gpt-3.5 and gpt-4 families of large language models (llms) and has been fine-tuned using both supervised and reinforcement learning techniques. it possesses diverse abilities of nlp, including question answering, dialogue generation, code debugging, generation evaluation @xcite @xcite @xcite . we are particularly interested in chatgpt for domain-specific machine translation (mt) tasks, especially how domain-specific knowledge can be introduced into chatgpt to improve the translation accuracy of domain entities. source: 甚至光纤上去一照变成黑光纤了。 mt by chatgpt: even the fiber op�c cable turned into a black fiber op�c cable when it was lit up. reference: even the op�cal fiber is irradiated to become a dark fiber. prompt for chatgpt: 请将下面的句子翻译成英文：甚至光纤上去一照变成黑光纤了。 figure 1: an mt example by chatgpt in the vlc domain, where 光纤 and 黑光纤 are two domain entities and the corresponding english translations are optical fiber and dark fiber respectively, and the prompt "请将下面的句子翻译成英文: ..." means "please translate the following sentence into english: ...". recent studies @xcite @xcite on translation tasks have found that chatgpt performs competitively with commercial translation products (e.g., google translate and microsoft translate) in the general domain, but performs poorly in specific domains. according to the analysis by @xcite and @xcite , it is a challenge for chatgpt to correctly translate domain entities. fig. 1 shows a chinese sentence that contains two domain entities 光纤 and 黑光纤 in the visible light communication (vlc) domain. we use the translation prompt in fig. 1 for chatgpt to translate this sentence into english. from fig. 1 , it could be seen that the two entities are wrongly translated as fiber optic cable and black fiber optic cable by chatgpt, and the correct translations are optical fiber and dark fiber. although chatgpt shows strong translation capabilities in general domains, it is not so effective in specific domains, especially in domain entity translation. in this paper, we propose to utilize multilingual knowledge graph (mkg) to help chat-gpt address this issue. after we introduce bilingual entity pairs from mkg into translation prompts, chatgpt can use the correct translations of domain entities. experimental results in three specific domains (biomedical science, vlc, and wireless) demonstrate that the overall translation quality of chatgpt is improved by +6.21, +3.13 and +11.25 respectively in terms of bleu scores, and the translation accuracy of domain entities is improved by +43.2%, +30.2% and +37.9% absolute points respectively. the main contributions of this paper are as follows: ## translation prompt with mkg the generation of mkg-based translation prompts for chatgpt is described in fig. 2 , which consists of three steps: source: 甚至光纤上去一照变成黑光纤了。 mt by chatgpt + mkg: even when light was shone on the op�cal fiber, it turned into dark fiber. reference: even the op�cal fiber is irradiated to become a dark fiber. mt by chatgpt: even the fiber op�c cable turned into a black fiber op�c cable when it was lit up. figure 3 : translation results of chatgpt with or without mkg for the sentence in fig. 2 . fig. 3 shows the translation results of chatgpt with or without mkg for a sentence in the vlc domain (i.e., using the prompts in fig. 2 and fig. 1 for chatgpt respectively). it could be seen that chatgpt correctly translates the two domain entities 光纤 and 黑光纤 with mkg, but incorrectly without mkg. this means that introducing mkg to translation prompts does improve the entity translation accuracy of chatgpt, thereby contributing to the translation quality of chatgpt. ## conclusion in this paper, a novel mkg-based translation prompt is designed for chatgpt to improve the domain-specific translation quality, especially the translation accuracy of domain entities. domain entities are first recognized from the source sentences and are used to extract corresponding bilingual entity pairs from mkg. then mkg-based translation prompts are generated for chatgpt with the bilingual entity pairs. experimental results in the three specific domains demonstrate that the chatgpt empowered by mkg-based translation prompts greatly improves the translation accuracy of the domain entities, even outperforming google translate. however, the bleu scores of chatgpt with the mkg-based translation prompts in two of the three domains are still lower than those of google translate, although they are greatly higher than those of the chatgpt without mkg. this suggests that better prompt engineering needs to be developed for chatgpt to further unlock its potential in translation. in addition, the performance of the mkg-based translation prompts for chatgpt in the general domain (news domain) is also investigated.