Guiding Neural Machine Translation with Semantic Kernels (2022)

Abstract: Machine Translation task has made great progress with the help of auto-regressive decoding paradigm and Transformer architecture. In this paradigm, though the encoder can obtain global source representations, the decoder can only use translation history to determine the current word. Previous promising works attempted to address this issue by applying a draft or a fixed-length semantic embedding as target-side global information. However, these methods either degrade model efficiency or show limitations in expressing semantics. Motivated by Functional Equivalence Theory, we extract several semantic kernels from a source sentence, each of which can express one semantic segment of the original sentence. Together, these semantic kernels can capture global semantic information, and we project them into target embedding space to guide target sentence generation. We further force our model to use semantic kernels at each decoding step through an adaptive mask algorithm. Empirical studies on various machine translation benchmarks show that our approach gains approximately an improvement of 1 BLEU score on most benchmarks over the Transformer baseline and about 1.7 times faster than previous works on average at inference time.

## introduction machine translation has been a long-standing task in natural language processing @xcite . recently, neural-based machine translation (nmt) models @xcite @xcite have made great progress and become the mainstream of machine translation frameworks. most nmt models adopt the encoder-decoder framework. the encoder transforms the source sentence into source-side global representations. and the decoder generates the target sentence auto-regressively, based on the sourceside representations and translation history. however, one limitation of such auto-regressive decoding is that the generation of word y t only has access to target-side partial information @xmath0. if translation history is mistranslated, this error will be propagated to all subsequent words @xcite . also, this makes the generation heavily dependent on the source sentence, and minor changes in source sentence may lead to dramatic degradation in translation outcome @xcite . intuitively, using target-side global information to guide translation progress can alleviate this problem. ## nmt with semantic kernels to make nmt model comply more with source sentence meaning than source sentence form, we propose skam, which consists of three modules: kernel selection module, kernel projection module, and decoding module, as depicted in figure 2 . we will explain each module in the following section. ## experiments we conduct experiments on the following benchmarks: nist chinese to english (@xmath1, wmt14 english to german (@xmath2, wmt14 english to french (@xmath3, iwslt14 english to/from german (en↔de) translation tasks. ## conclusion following functional equivalence theory, we propose semantic kernels with adaptive decoding, which extracts several semantic kernels and projects them into target embedding space to guide translation. we propose adaptive mask mechanism to enable each decoding step to access target-side global information. several empirical results reveal that our skam is both expressive in semantics and efficient in time. our way of representing kernel sentences in nmt is intuitive and simple. in future work, we would like to explore better methods to capture sentence semantics.