Comparison of the effects of attention mechanism on translation tasks of different lengths of ambiguous words (2020)

Abstract: In recent years, attention mechanism has been widely used in various neural machine translation tasks based on encoder decoder. This paper focuses on the performance of encoder decoder attention mechanism in word sense disambiguation task with different text length, trying to find out the influence of context marker on attention mechanism in word sense disambiguation task. We hypothesize that attention mechanisms have similar performance when translating texts of different lengths. Our conclusion is that the alignment effect of attention mechanism is magnified in short text translation tasks with ambiguous nouns, while the effect of attention mechanism is far less than expected in long-text tasks, which means that attention mechanism is not the main mechanism for NMT model to feed WSD to integrate context information. This may mean that attention mechanism pays more attention to ambiguous nouns than context markers. The experimental results show that with the increase of text length, the performance of NMT model using attention mechanism will gradually decline.

## introduction natural language always contains many ambiguous words. ambiguous words mean that the same word can express many different meanings in different contexts. since the actual meaning of ambiguous words is closely related to context information and context, it is always a challenge to machine translates sentences containing ambiguous words. in statistical machine translation (smt) system, we can improve the translation effect of ambiguous words by taking the context marker of ambiguous words into account. in the neural machine translation (nmt) @xcite #:they have the same contribution to the article. task, especially in the nmt @xcite model which uses attention mechanism, the hidden state of each input contains context information. therefore, theoretically, the attention mechanism can make the nmt model better deal with the translation task of ambiguous words. however, there are no empirical results to show that attention mechanism can obtain enough information of word sense disambiguation from hidden state. moreover, we still don't know the deep principle of the attention mechanism to deal with ambiguous words. this paper mainly studies whether the nmt model including attention mechanism will have different performance when dealing with different states of ambiguous word text. we compared the performance of different nmt models using attention mechanism in different length texts with ambiguous words, and evaluated them with bleu. in the following chapters, we use different nmt models with attention mechanism to test the long text and short text containing ambiguous words, and compare the influence of the length of the text containing ambiguous words on the translation effect. our findings are summarized as follows: this paper mainly studies the performance of attention mechanism on word sense disambiguation in different length texts. more specifically, we explore whether attention plays a better role in long text word sense disambiguation tasks as well as in short text tasks or not. this paper mainly studies the performance of attention mechanism on word sense disambiguation in different length texts. more specifically, we explore whether the role of attention in long text translation is as good as we expected or not. ## evaluation in this paper, nmt model is used to evaluate the translation of data sets with different text length. we evaluate two popular nmt models with different attention mechanisms, one is google t5 with advanced attention mechanism, the other is martin mt using ordinary attention mechanism (tiedemann and thottingal, 2020). we use two different data sets for comparative test, one is the ordinary opus parallel corpus, and the other is the screened corpus data set containing ambiguous words. the original text length of the data set containing ambiguous nouns ranges from three words to eighty words. in order to ensure the differentiation of nmt model in short text task and long text task, we randomly extract 2000 texts from the test data set with less than 20 words and the test data set with 40 to 80 words as the comparison data set the results were translated by nmt model and evaluated by bleu. ## results ## analysis through the data, it can be found that the average performance score of two nmt models with attention mechanism on ambiguous data sets is not as good as that of ordinary data sets, which proves that the text with ambiguous words does affect the performance of machine translation tasks. that is to say, the analysis of the performance of the nmt model with attention mechanism on ambiguous text is meaningful. by comparing the performance of nmt model on different length texts, we can get the following information. first of all, the different text length on ordinary opus dataset has no obvious effect on nmt model translation performance. on the contrawsd dataset with ambiguous words, it is obvious that the performance of nmt model in the translation task of ambiguous text tends to decline with the increase of text length. next, we explore the details of the attention weight when dealing with ambiguous words. due to different text scales, we use a residual coefficient to express the attention weight matrix. the specific calculation method is as follows: where n is the length of the text, e is the diagonal matrix of size n, and wi is the attention weight matrix of layer i. the larger the residual coefficient r is, the less attention weight is. then we test the variation trend of r under different text lengths on the contrawsd dataset containing ambiguous word text, and get the results shown in figure 2 . the results show that with the increase of text length, the performance of the attention weight gradually decreases, which just confirms that the performance of the nmt model with attention mechanism decreases with the increase of the length of the text. ## conclusion this paper analyzes the performance of the attention mechanism in nmt for different length texts with ambiguous words. we use opus and contrawsd as test sets to evaluate the translation of nmt model under different length texts. the results showed that the bleu scores of t5 model on short text and long text were 26.078 and 7.877, while those of martin mt model were 25.323 and 12.479, respectively. the sparsity of ambiguous words in the training set may be the main problem leading to incorrect translation. however, the attention mechanism does perform better on short text data sets than on long text data sets. this is probably because the fact that the attention mechanism tends to pay more attention to the ambiguous words themselves than to the context markers. it is significant to understand the performance differences of attention mechanism on different scale text data sets, which may indicate the potential optimization direction of attention mechanism in such tasks. we hope that our future work can continue to improve our understanding of the deepseated principle of attention mechanism in nmt model, analyze the details of attention weight in nmt model when dealing with ambiguous word text tasks, and explore how to improve the translation effect of nmt model in different scale text sense disambiguation tasks.