Transformer-Based Direct Hidden M arkov Model for Machine Translation (2021)

Abstract: The neural hidden Markov model has been proposed as an alternative to attention mechanism in machine translation with recurrent neural networks. However, since the introduction of the transformer models, its performance has been surpassed. This work proposes to introduce the concept of the hidden Markov model to the transformer architecture, which outperforms the transformer baseline. Interestingly, we find that the zero-order model already provides promising performance, giving it an edge compared to a model with first-order dependency, which performs similarly but is significantly slower in training and decoding.

## introduction recently, significant improvements have been made to neural machine translations (nmt). regardless of whether a recurrent neural network with long short-term memory (hochreiter and schmidhuber, 1997) (lstm-rnn) @xcite or a convolutional neural network (cnn) @xcite or a self-attentive transformer network @xcite is used, the attention mechanism is always one of the key components that all state-of-the-art nmt systems contain. several attempts have been made to explore alternative architectures that do not use an attention mechanism @xcite @xcite . however, either the performance of those systems is significantly worse than that of the lstm-rnn-based approaches, or the time and memory complexity is much higher. since the transformer architecture has upgraded the state-of-the-art to an even higher standard, fewer studies are being carried out in this direction. despite the promising translation performance of the transformer architecture, recent studies have found that the quality of the word alignments produced by the multi-head cross-attention weights is quite poor, and various techniques are proposed to address this problem @xcite @xcite . while these works focus on extracting promising alignment information from the transformer architecture, we aim to improve the translation performance of the baseline model by introducing alignment components while keeping the system monolithic. to this end, the possibilities are studied to apply the transformer architecture to the direct hidden markov model (hmm), which is not as straightforward as in the case of lstm-rnn due to the cross-attention through all decoder layers. experimental results show that the zero-order direct hmm already outperforms the baseline transformer model in terms of ter scores @xcite , while the first-order dependency with higher computational complexity offers no further improvements. ## related work the attention component is introduced by @xcite in nmt to simulate the alignment between the source and target sentence, which leads to significant improvements compared to the pure sequence-to-sequence model @xcite . @xcite present a lstm-rnnbased hmm that does not employ an attention mechanism. this work aims to build a similar model with the transformer architecture. while they perform comparable to the lstm-rnn-based attention baseline with a much slower model, our model outperforms the transformer baseline in terms of ter scores. the derivation of neural models for translation on the basis of the hmm framework is also studied in @xcite and @xcite . in @xcite , alignment-based neural models are used to model alignment and translation from the target to the source side (inverse direction), and a language model is included in addition. and @xcite rely on alignments generated by statistical systems that serve as supervision for the training of the neural systems. by contrast, the model proposed in this work does not require any additional language model or alignment information and thus keeps the entire system monolithic. several works have been carried out to change attention models to capture more complex dependencies. @xcite introduce structural biases from word-based alignment concepts such as fertility and markov conditioning. @xcite incorporate lexical probabilities to influence attention. these changes are based on the lstm-rnn-based attention model. @xcite and @xcite try to generate translation and high-quality alignment jointly using an end-to-end neural training pipeline. by contrast, our work focuses more on improving the translation quality using the alignment information generated by the self-contained model. ## direct hmm the goal of machine translation is to find the target language sentence e i 1 = e 1 , e 2 , • • • , e i that is the translation of a particular source language sentence f j 1 = f 1 , f 2 , • • • , f j with the maximum likelihood (arg max i,e i 1 pr(e i 1 |f j 1 ) ). in the direct hmm, an alignment from target to source (@xmath0: the term "direct" refers to the modeling of p(e|f ) instead of p(f |e) as in the conventional hmm @xcite . in @xcite , two lstm-rnn based neural networks are used to model the lexicon and the alignment probability separately. in this work they are modeled with a single transformer-based network. ## direct hmm in transformer this section describes in detail how we modify the transformer model so that both the alignment and the lexicon probability can be generated. while the lexicon model in the direct hmm has a zero-order dependency on the current alignment position b i : pr(e i |b i 0 , e i-1 0 , f j 1 ) := p(e i |b i , e i-1 0 , f j 1 ) (4) we implement zero-and first-order dependencies for the alignment model. ## conclusion this work exhibits the use of the transformer architecture in a direct hmm for machine translation, which significantly improves ter scores. in addition, we show that the proposed system tends to "refuse" to learn first-order dependency during training. the zero-order model achieves a good compromise between performance and decoding speed, which is much faster than previous work on the direct hmm. in order to benefit from the predecessor alignment information, further techniques should be carried out. another future work would be to combine the attention mechanism with the alignment information to further improve performance.