Memory Attentive Fusion: External Language Model Integration for Transformer-based Sequence-to-Sequence Model (2020)

Abstract: This paper presents a novel fusion method for integrating an external language model (LM) into the Transformer based sequence-to-sequence (seq2seq) model. While paired data are basically required to train the seq2seq model, the external LM can be trained with only unpaired data. Thus, it is important to leverage memorized knowledge in the external LM for building the seq2seq model, since it is hard to prepare a large amount of paired data. However, the existing fusion methods assume that the LM is integrated with recurrent neural network-based seq2seq models instead of the Transformer. Therefore, this paper proposes a fusion method that can explicitly utilize network structures in the Transformer. The proposed method, called memory attentive fusion, leverages the Transformer-style attention mechanism that repeats source-target attention in a multi-hop manner for reading the memorized knowledge in the LM. Our experiments on two text-style conversion tasks demonstrate that the proposed method performs better than conventional fusion methods.

## introduction in recent studies, the transformer sequence-tosequence (seq2seq) model @xcite has successfully performed in various natural language generation tasks, such as machine translation @xcite , image captioning @xcite @xcite , and automatic speech recognition @xcite @xcite . although the transformer training needs paired data, a large amount of paired data cannot often be prepared. moreover, unpaired data cannot be used for training the transformer even though such data can be collected on a large scale. to utilize a large amount of unpaired data, methods of integrating an external language model (lm) trained with these data into seq2seq mod-els have been proposed @xcite @xcite . these methods can improve the fluency of sentences that are generated by seq2seq models; however, they integrate the lm into recurrent neural network (rnn) based seq2seq models rather than the transformer. in other words, lm fusion methods specific to the transformer have not been considered yet. here, the transformer employs the multi-hop attention mechanism @xcite that repeats the source-target attention mechanism in each transformer decoder block. thus, it is supposed that the source-target attention mechanism promotes to extract effective source information for target tasks more exactly than rnn based seq2seq models. therefore, we assume that the transformer can utilize memorized knowledge in the external lm more effectively by using the multi-hop attention mechanism for the lm fusion. in this paper, we propose a novel fusion method, called memory attentive fusion, to integrate an external lm into the transformer. this fusion method utilizes a multi-hop source-target attention mechanism for combining the transformer decoder with the external lm. we performed experiments with two text-style conversion tasks: spoken-to-written style conversion and dialect conversion. our experiments demonstrate that the proposed method performs better than conventional fusion methods. ## related work the simplest fusion method is to train the seq2seq model and the lm independently and then combine their outputs @xcite @xcite . these methods are called shallow fusion. moreover, methods that integrate an external lm into seq2seq models during training have been proposed: deep fusion @xcite and cold fusion @xcite . these methods utilize the information of not only paired data but also unpaired data in training. figure 1 shows a transformer with cold fusion. these methods assume that the lm is integrated into rnn-based seq2seq models instead of the transformer. ## memory attentive fusion this section details memory attentive fusion for the transformer seq2seq model. in fact, memory attentive fusion is an extended method of the cold fusion @xcite . while the cold fusion uses memorized knowledge in the lm at an output layer only once, the memory attentive fusion repeatedly uses the knowledge at transformer decoder blocks based on a source-target attention mechanism. we define an input sequence as @xmath0, • • • , x m } and an output sequence as @xmath1, • • • , y n }, where x m and y n are tokens in the input and output sequence. in text-style conversion, the model predicts the generation probabilities of the output sequence given the input sequence. the generation probability of y is defined as encoder: the encoder converts an input sequence x into the hidden representations s (k) using k transformer encoder blocks. first, the input hidden representation of the transformer encoder block figure 1: transformer with cold fusion. figure 2: transformer with memory attentive fusion. decoder with memory attentive fusion: the decoder with memory attentive fusion computes the generation probability of a token from the preceding tokens and hidden representations of the input sequence and the lm information. the predicted probabilities of the n-th token y n are calculated as this attention mechanism is repeated with transformer decoder block in the multi-hop manner. therefore, we expect to read the memorized memory in the lm effectively. next, we concatenate the hidden vector that have target and source information, and that have target and the lm information with gating mechanism by training: in the transformer, the model parameter set can be optimized from training dataset note that the external lm uses the freezing parameter θ lm . ## experiments we evaluated our method on text-style conversion tasks. in particular, we chose spoken-to-written style conversion task and dialect conversion task in japanese. in the spoken-to-written style conversion task, spoken-style text produced by an automatic speech recognition system is converted into written-style text that has correct punctuation and no disfluency @xcite . in the dialect conversion task, japanese dialects are converted into standard japanese. ## conclusion we proposed memory attentive fusion, a novel method to integrate an external lm into the transformer. conventional fusion methods assume that the lm is integrated into the rnn-based seq2seq. on the other hand, the proposed method employs a transformer-specific fusion method which repeats the attention mechanism for the lm many times. experiments demonstrated that the proposed method outperformed the conventional methods in two tasks. we conclude that the proposed method is suitable for integrating the lm into the transformer. in the future work, we will try using the proposed method in other natural language generation tasks such as automatic speech recognition.