Content Word Aware Neural Machine Translation (2020)

Abstract: Neural machine translation (NMT) encodes the source sentence in a universal way to generate the target sentence word-by-word. However, NMT does not consider the importance of word in the sentence meaning, for example, some words (i.e., content words) express more important meaning than others (i.e., function words). To address this limitation, we first utilize word frequency information to distinguish between content and function words in a sentence, and then design a content word-aware NMT to improve translation performance. Empirical results on the WMT14 English-to-German, WMT14 English-to-French, and WMT17 Chinese-to-English translation tasks show that the proposed methods can significantly improve the performance of Transformer-based NMT.

## introduction neural machine translation (nmt) models @xcite @xcite often utilize the global neural networks to encode all words for learning the sentence representation and the context vector, and computes the accuracy of each generated target word in a universal manner. meanwhile, each generated target word makes the same contribution to the optimization of the nmt model, regardless of its importance. actually, there lacks a mechanism to guarantee that nmt captures the information related to word importance when predicting translations. intuitively, content words express more important meanings than function words, which indicates their comparative significance. to evaluate this, we randomly masked content or function words with unk in a source sentence. figure 1 shows that the bleu scores of the test set decreased much more substantially when parts of content words were randomly replaced with unk on the wmt14 english-to-german task, which is in line with the findings in @xcite 's work. to address this limitation, we propose a content word-aware nmt model that exploits the results of translation using a sequence of content words learned by a simple content word recognition method. inspired by the works of @xcite @xcite , we first divide words in a sentence into content words and other function words depending on term frequencyinverse document frequency (tf-idf) constraints. two methods are designed to utilize the sequence of content word on the source and target sides: 1) we encode the content words of the source sentence as a new source representation, and learn an additional content word context vector based on it to improve translation performance; 2) a specific loss for content words of the target sentence is introduced to compensate for the original training objection, to obtain a content word-aware nmt model. empirical results on the wmt14 english-to-german, wmt14 englishto-french, and wmt17 chinese-to-english tasks show the effectiveness of the proposed method. ## background: transformer-based nmt in transformer-based nmt @xcite , the encoder is composed of a stack of l identical layers, each of which contains two sub-layers. the first sub-layer is a self-attention module (att), and the second sub-layer is a position-wise fully connected feed-forward network (fnn). a residual connection @xcite is applied between the sub-layers, and layer normalization (ln) @xcite is performed. formally, the l-th identical layer of this stack is as follows: similarly, the decoder is composed of a stack of l identical layers. compared with the stacked encoder, it contains an additional attention sublayer to compute alignment weights for the output of the encoder stack h l : we explore the effects of content words in a sentence for nmt. specifically, we propose a content word recognition method based on the tf-idf @xcite ). an input sentence of length j m is treated as a document d m , and the tf-idf t i j for each word d j in d m is computed: ## content word aware nmt in this section, we propose two ways to make use of the information on content words, designing three content word-aware nmt models. the proposed method of content word recognition is first added as an additional module to the encoder to learn the sequence of source content words x from the input source sentence. x is mapped and fed into the shared encoder @xcite in eq.( 1 ) to learn an additional source representation of content words h l . an multi-head attention module is then introduced to the decoder to learn the context vector c l i based content words at time-step i, and c l i is used to enhance the output s l i : lx target output embedding multi-head attention feed forward lx source input embedding linear softmax output probabilities add & norm add & norm add & norm feed forward add & norm add & norm multi-head attention masked multi-head attention positional encoding positional encoding positional encoding shared encoder multi-head attention target 𝐿(𝐲|𝐱; 𝜃) add & norm feed forward add & norm multi-head attention content word recognition loss (a) lx target output embedding multi-head attention feed forward lx source input embedding linear softmax output probabilities add & norm add & norm add & norm feed forward add & norm add & norm positional encoding multi-head attention masked multi-head attention note that both the original source representation h l and proposed content word based representation h l are learned by a shared encoder using our content word recognition module. ## conclusion and future works this paper explored the importance of word for nmt. we divided words of one sentence into content and function words through word frequency-related information. our proposed nmt models, that are easy to implement and not much time and space cost, are introduced to the training and inference, and can improve the representation and translation of content words. in future work, we will investigate the impact of fine-grained word categories (such as nouns, verbs, and adjectives) on the translation performance and design specific methods according to these categories.