Neural Machine Translation with Heterogeneous Topic Knowledge Embeddings (2021)

Abstract: Neural Machine Translation (NMT) has shown a strong ability to utilize local context to disambiguate the meaning of words. However, it remains a challenge for NMT to leverage broader context information like topics. In this paper, we propose heterogeneous ways of embedding topic information at the sentence level into an NMT model to improve translation performance. Specifically, the topic information can be incorporated as pre-encoder topic embedding, post-encoder topic embedding, and decoder topic embedding to increase the likelihood of selecting target words from the same topic of the source sentence. Experimental results show that NMT models with the proposed topic knowledge embedding outperform the baselines on the English -> German and English -> French translation tasks.

## introduction neural machine translation (nmt) utilizes local context captured from the mapping between the bitexts to disambiguate the meaning of words. while existing nmt models can handle meaning ambiguities based on local contexts learned from explicit collocations, it remains a challenge for nmt to produce accurate results for words presented in implicit collocations. the notion of implicit collocation is referred to as the circumstance when the meaning of two or more words can not be learned from the available training data; broader context information like topics may be utilized to generate an accurate meaning. for example, in the sentence "he likes bank fishing", the word "bank fishing" incorporating topic information into nmt has been explored in @xcite and @xcite with both studies adapting latent dirichlet allocation (lda) @xcite to model topics of source and target languages. both works utilized the traditional encoder-decoder architecture with gated recurrent units (gru) @xcite . although @xcite showed that topic knowledge incorporation is also applicable to the transformer architecture @xcite , it is argued that the joint learning of topic modeling and nmt is not an ideal way. the training of topic models can leverage a large volume of easily accessible monolingual data. once a topic model is learned, it can be reused in different translation scenarios without retraining nmt models. therefore decoupling topic modeling and nmt is a more flexible and scalable option. in this paper, we propose heterogeneous ways of incorporating topic information into the transformer architecture. specifically, the topic information can be incorporated in a heterogeneous manner, namely as pre-encoder topic embedding (en c pre ), post-encoder topic embedding (en c post ), and decoder topic embedding (dec). besides, the topic distribution learned for each word (as its topic embedding) is summarized at the sentence level and fed into the nmt model. the intuition is that aggregating topic distribution at the sentence level produces more accurate topic information than at the word level. this enables topic modeling to consider contexts conveyed in a sentence. each target word is generated with the guidance of the topic information of both source and target sentences. the topic-enhanced nmt models are trained on wmt14 @xmath0. experimental results show that our approach can significantly improve translation quality with the topic embedding by achieving up to +1.57 bleu score improvement over the transformer baselines. the effect of the proposed method is also verified on the @xmath1. ## related work many studies have focused on using topic information as explicit prior knowledge to help model learn sentence representations on nlp tasks, such as @xcite ; @xcite ; @xcite . topic modeling has shown its effectiveness in statistical machine translation (smt) models @xcite @xcite . incorporating topic information into nmt has recently been explored by @xcite al. (2016); zhang et al. (2016); wei et al. (2019); chen et al. (2019). zhang et al. (2016 ) proposed a topicinformed nmt model leveraging source-side and target-side topics, separately learned by two independent lda models from training data. wei et al. (2019) designed a bilingual topic nmt model incorporating bilingual topic knowledge into nmt to improve translation performance. both works were built upon gated recurrent units (gru) architecture with limited coverage to the transformer architecture. both studies adopted lda to model topics of source and target languages. dieng et al. (2020) pointed out that lda is not an effective learner for data with an extensive vocabulary because one has to remove the most and least frequent words to fit good topic models. this pruning practice limits the scope of lda models. the embedding topic model (etm) @xcite ) was proposed to model each term as an embedding and each topic as a point in that embedding space. the per-topic conditional probability of a term has a log-linear form to preserve low-dimensional representation of the vocabulary so that etm can discover interpretable topics with large vocabularies, including rare words and stop words. in this study, we apply etm to handle issues associated with large vocabularies. @xcite used a variant of convolutional neural networks (cnn) to learn latent topic representations implicitly from sentence-level context. an additional multi-head attention module is directly involved in learning the attentions between topics and targeting words independently from the encoding of the transformer. @xcite also tried an explicit topic representation computed by tf-idf, but did not perform better than their latent version. in this paper, we propose multiple heterogeneous ways of explicitly integrating topic information into nmt, resulting in better performance. the topic knowledge in the figure is obtained from the topic embedding tables for source and target languages produced by etm. ## experiments datasets embedding topic model: we use the 20 newsgroups corpus for training the english embedding table and use the wmt14 german monolingual dataset to train the german embedding table. to verify the effect of the proposed method on the @xmath2, we sample only 10 million (m) sentences representing less than one third of the training data randomly from wmt14 french monolingual dataset to train the french embedding table . the experiments are conducted on the standard wmt 14 @xmath3. we evaluate the models on the newstest 2014, while the concatena- training details we preprocess the corpus for all experiments of etm. we set the number of the topics to 50 and epoch number to 500, which are empirical values adopted from etm. after preprocessing, we further remove one-word documents from the validation and test sets. for all nmt experiments, we train our models on one machine with 4 nvidia v100 gpus and follow @xcite base model to set the hyper-parameters with model configurations. the number of parameters is 129m. we compare our topic model against the following models: fairseq (base) is a sequence modeling toolkit @xcite . blt-nmt is a topic enhanced model with incorporated bilingual topic knowledge into nmt @xcite . ltr-nmt is a topic-based nmt model using a cnn model @xcite . ## conclusion in this paper, we propose heterogeneous ways of incorporating topic information as prior knowledge into the transformer architecture to improve trans-example 1 source since 2010, johanna rahner has occupied a chair systematic theology in the institute for catholic theology at the university of kassel. target seit 2010 hat johanna rahner einen lehrstuhl für systematische theologie am institut für katholische theologie der universität kassel inne. base nmt johanna rahner hat seit 2010 eine lehrstuhl für systemtheorie am institut für katholische theologie der universität kassel inne. topic nmt seit 2010 hat johanna rahner einen lehrstuhl für systematische theologie am institut für katholische theologie der universität kassel inne. example 2 source an obama voter's cry of despair. target verzweiflungsschrei eines obama wählers. base nmt obamas ruf der verzweiflung. topic nmt der schrei der wähler eines obama. example 3 source the previous silence was indeed a reaction to the events of previous days. target das schweigen zuvor war wohl eine reaktion auf die geschehnisse der vergangenen tage. base nmt das vorherige schweigen war in der tat eine reaktion auf die ereignisse der vortage. topic nmt das vorhergehende schweigen war in der tat eine reaktion auf die ereignisse der vergangenen tage.