Interpreting Positional Information in Perspective of Word Order (2023)

Abstract: The attention mechanism is a powerful and effective method utilized in natural language processing. However, it has been observed that this method is insensitive to positional information. Although several studies have attempted to improve positional encoding and investigate the influence of word order perturbation, it remains unclear how positional encoding impacts NLP models from the perspective of word order. In this paper, we aim to shed light on this problem by analyzing the working mechanism of the attention module and investigating the root cause of its inability to encode positional information. Our hypothesis is that the insensitivity can be attributed to the weight sum operation utilized in the attention module. To verify this hypothesis, we propose a novel weight concatenation operation and evaluate its efficacy in neural machine translation tasks. Our enhanced experimental results not only reveal that the proposed operation can effectively encode positional information but also confirm our hypothesis.

## introduction in recent years, attention mechanism @xcite @xcite has made remarkable progress on a wide range of natural language processing (nlp) tasks, such as machine translation @xcite , question answering and language inference @xcite . it updates the contextual representation of a word by aggregating information from other words in the context, enabling it capture the content-based relevance of any two words, regardless of the distance between them. in contrast to widely used recurrent neural networks (rnns) or convolutional neural networks (cnns), attention mechanism suffers from a notable disadvantage, i.e. its permutation invariance @xcite . this limitation results in its a i is the attention score vector of x i over other words and v is the value matrix. ⊙, , ⊕ are scalar multiplication, vector sum and vector concatenation respectively. the weight concatenation combines weighted elements in accordance with the word order and linearly transforms the output to its original dimensions. this process serves to establish positional dependencies throughout a given sequence. besides, previous studies have probed the influence of word order perturbation on natural lan-guage understanding (nlu) tasks @xcite @xcite to determine whether these tasks are sensitive to word order. the findings indicate that some nlu tasks do require word order information although others do not. it should also be noted that word order is particularly important for natural language generating (nlg) tasks, such as machine translation. this is because metrics used to evaluate generated results, such as bleu @xcite , are sensitive to word order. therefore, word order is a critical aspect of natural language processing. in linguistics, grammars can be viewed as explanations for rules or principles of word order @xcite to some extent. given a sentence of arbitrary length, a set of positional embeddings can be seen as a group of distributed vectors that represent temporal information @xcite , namely word order in the context of language processing. this insight motivates us to correlate positional information with word order, interpret positional information in perspective of word order, and comprehend how word order impacts attention models. in this paper, we first examine how the attention module of transformer works and discover that its weight sum operation (figure 1(a) ) is the reason it struggles to encode positional information. furthermore, this sheds light on how the word order impacts on nlp models, which means that the impact is achieved by its connection with positional information. based on this finding, we make further modifications to the attention module's working mechanism. our goal is to implicitly encode positional information in attention module, which we posit is superior to explicit representation of positional information, as suggested by @xcite . to be more concrete, we devise a novel weight concatenation operation (figure 1(b) ) to calculate the final contextual representation, as an alternative to the weight sum operation (figure 1(a) ) in the attention module. to test the effectiveness of this approach, we evaluate the novel operation on the widely used, big neural machine translation datasets, including wmt 2014 english⇒german and wmt 2014 english⇒french. our experimental results demonstrate that the proposed operation is capable of effectively encoding positional information for a sequence, leading to consistently improved performance and verifying our hypothesis. ## discussion effect of positional kernel's dimension as stated previously in equation ( 22 ), the positional kernel tensor φ has the shape of l × d 1 × d 2 . in practice, we use d 1 = d 2 = d p . to further investigate the effect of the dimension d p , we conduct experiments on posnet-embed with a series of varying d p and evaluate the performance of dif- for the transformer big model, which is just twice the size of the base model. evaluation with different n-grams the default evaluation setup is 6-gram character f2 and 2gram word f2 for chrf++, 4-gram word precision for bleu. however, since the posnet preserves positional information through concatenating the weighted value vectors, it is intriguing what the performance is when evaluating with coarser-grain unit. therefore, to observe the performance superiority of the proposed method on coarser grain, we evaluate the performance gap of posnet-embed over sape on wmt14 en-de with different ngrams. specifically, we test the chrf with n-gram character f2 and 2-gram word f2 (1 ≤ @xmath0, the chrf with 6-gram character f2 and n-gram word f2 (1 ≤ @xmath1, and the bleu with ngram word precision (1 ≤ @xmath2. as shown in figure 4 , it is amazing that the performance superiority of the posnet-embed over the sape consistently enlarges as the evaluating grain becomes coarser, i.e. bigger n-gram character or word. we speculate that this is attributable to the weight concatenation of posnet since the concatenation operation promotes the model to capture the whole context precisely according to the word order of a sequence, which preserves the coarser-grained feature. ## related work since transformer relies solely on the attention mechanism, it is position-insensitive to sequences with different word orders in the absence of positional encodings. many studies attempt to utilize various positional encodings to inject the sequence's positional information into the model. typically, embedding-level positional encodings @xcite @xcite build positional dependencies via adding positional encodings to embedding results. in addition, attention-level positional encodings @xcite @xcite usually capture positional information based on an analysis of attention mechanisms and leverage delicate indexing manners, such as the relative distance to the querying word. however, it is unknown why the attention mechanism of the transformer is position agnostic and what the working principle behind these positional encodings is. there has been increasing interest in understanding and explaining how these positional encodings construct positional dependencies for a sequence. the proof in @xcite suggests that the key of eliminating transformer's permutation invariance is to quantify the embedding results to distinct intervals, which ensures a one-to-one mapping. @xcite analysed and improved the scalar relative positional encoding of t5 @xcite ) from a probabilistic perspective and took it as a prior distribution. the difference is that, in this work, we associate the positional information with the word order of a sequence, providing a novel linguistic perspective on positional encoding. furthermore, we propose a new and effective positional encoding mechanism. we perform the comparison of different positional encoding methods and the results are in tables 1 and 2 except that the theoretical positional encoding scheme in the proof of yun et al. (2020) doesn't work well in practice and thus are not reported. ## conclusion positional encoding is a critical issue for transformer-like models. however, it has not been explored how positional encoding builds positional dependencies for a sequence. in this paper, we analyse the working manner of the attention module and find that its weight sum operation leads to the failure to encode positional information. then, we modify the attention module with a proposed novel weight concatenation operation, following the guideline of retaining positional information according to the word order of a sentence. the modification correlates a sequence's word order with positional information, thus shedding light on how the word order impacts on nlp models. competitive experimental results substantiate our hypothesis.