Knowledge Graph Enhanced Neural Machine Translation via Multi-task Learning on Sub-entity Granularity (2020)

Abstract: Previous studies combining knowledge graph (KG) with neural machine translation (NMT) have two problems: i) Knowledge under-utilization: they only focus on the entities that appear in both KG and training sentence pairs, making much knowledge in KG unable to be fully utilized. ii) Granularity mismatch: the current KG methods utilize the entity as the basic granularity, while NMT utilizes the sub-word as the granularity, making the KG different to be utilized in NMT. To alleviate above problems, we propose a multi-task learning method on sub-entity granularity. Specifically, we first split the entities in KG and sentence pairs into sub-entity granularity by using joint BPE. Then we utilize the multi-task learning to combine the machine translation task and knowledge reasoning task. The extensive experiments on various translation tasks have demonstrated that our method significantly outperforms the baseline models in both translation quality and handling the entities.

## introduction neural machine translation (nmt) based on the encoder-decoder architecture becomes a new state-ofthe-art approach due to its distributed representation and end-to-end learning @xcite @xcite . during translation, the translation quality of the entities in a sentence has a great influence on the translation quality of the whole sentence. however, translating these entities remains challenging @xcite and various methods are proposed to improve the translation of entities @xcite @xcite . among them, some methods aim at incorporating the knowledge graph (kg) to utilize their structured knowledge on entities and improve the entity translation. these studies utilize kg to enhance the semantic representing of entities in a sentence @xcite or extract the important semantic vectors with kg @xcite . although great efforts have been made to incorporate kg into nmt, we find the existing methods have the following two problems: knowledge under-utilization: given a kg (denoted by k) and a parallel sentence pair dataset (denoted by d), the full entity set u can be divided into four subsets as shown in fig. 1 (a) : 1) k ∩ d entities, which appear in both k and d; 2) d -k entities, which only appear in d; 3) k -d entities, which only appear in k; 4) u -(k ∪d) entities, which are neither in k nor d. while previous studies @xcite @xcite only focus on the k ∩ d entities, the other three subsets are ignored. consequently, much knowledge on the other three subsets in kg is wasted. granularity mismatch: the current kg methods, such as knowledge embedding methods @xcite and knowledge reasoning methods @xcite , always utilize the entity as the basic granularity. while the nmt models utilize the sub-word @xcite or hybrid word-character @xcite as the translation granularity. this granularity mismatch between kg and nmt makes the current kg methods different to be utilized into nmt. shows an example that if we split a u -(k ∪d) entity "yansuananfeilatong" into fine granularity "yansuan", "anfei", and "latong", each fine granularity component can be found in k and d. therefore, to address above two problems, we propose a multi-task learning method on sub-entity granularity to combine kg and nmt. our main idea is to split the entities in k and d into fine granularity, and then enhance the semantic representation on this fine granularity. by doing so, we can achieve the following two goals: i) it can alleviate the knowledge under-utilization problem. fig. 1 (b) gives an example, where source entity "yansuananfeilatong" foot_0 is u -(k ∪ d) entity, which is ignored by previous studies. while when we split this entity into fine granularity "yansuan", "anfei" , and "latong", we find that each fine granularity component can be found in both k and d. therefore, we can enhance the semantic representation of these fine granularity components jointly with k and d. 2) the proposed method can also alleviate the granularity mismatch problem. it can enable both kg and nmt utilize an unified granularity (sub-entity), making knowledge in kg more easier to be utilized by the nmt. specifically, our proposed method contains two steps: i) joint bpe, we utilize the byte pair encoding (bpe) method @xcite to jointly split the entities in k and d into sub-entity granularity. ii) multi-task learning, we utilize multi-task learning to enhance the sub-entity embedding and the parameters in the neural model by machine translation task and knowledge reasoning task. in this step, we consider three different scenarios: a) only source kg is available; b) only target kg is available and c) both source and target kgs are available while they are non-parallel. the experimental results show that our proposed methods could improve the strong baseline models in translation quality, especially in handling the entities. generally, we make the following contributions in this paper: ## problem definition we denote the parallel sentence pair dataset by @xmath0, y )}, where x is the source sentence and y is the target sentence. besides d, we also need knowledge graph to improve the neural model. considering that it is difficult to obtain the parallel kgs. even in some low resource languages or domains, monolingual kg is also unavailable. therefore, we consider the following three scenarios: a) only source kg is available, b) only target kg is available, and c) both source and target kgs are available, while they are non-parallel. in this paper, we aim to improve the nmt model in these three scenarios. we denote the source kg by k @xmath1, r s , t s )}, where h s , t s and r s denote the head entity, tail entity and relation in source language, respectively. we denote the target kg by k @xmath2, r t , t t ) , where h t , t t and r t denote the head entity, tail entity and relation in target language, respectively. ## method description we propose a knowledge graph enhanced nmt via multi-task learning on sub-entity granularity, which contains two steps: i) joint bpe, and ii) multi-task learning. next, we will introduce each step in the following subsections. ## experimental setting training and evaluation details. we implement the nmt model based on the thumt toolkit foot_5 @xcite . we use the "base" version parameters of the transformer model. on all translation tasks, we use the bpe @xcite ) method to merge 30k steps. we evaluate the final translation quality with case-insensitive bleu @xcite for all translation tasks. ## related work the related work can be divided into two categories and we describe each of them as follows: knowledge graph in nmt. @xcite summarize the early studies using the knowledge graph or semantic web in statistical machine translation framework. recently, several studies incorporate the kg into nmt, where @xcite propose a knowledge-based semantic embedding for nmt by extracting the important semantic vectors with kg. @xcite incorporate kg by adding the relation constraint between the entities in the sentences. @xcite exploit the entity linking to disambiguate the entities found in a sentence. while these studies only focus on the k ∩ d entities. recently, @xcite utilize the entity alignment methods to improve the d -k entities. different from these methods, the proposed methods utilize multi-task learning on sub-entity granularity to make full use of kg and improve the entity translation. ## conclusion to improve the entity translation and make1 full use of kg, in this paper we propose a kg enhanced nmt method with multi-task learning on sub-entity granularity. we first represent the entity in kg and parallel sentence pairs into sub-entity granularity. then we utilize multi-task learning to improve the semantic represent of sub-entity and parameters in encoder or decoder. the proposed method can be utilized in different scenarios that source or/and target kg are available. the extensive experiments on