A Morphology-Based Investigation of Positional Encodings (2024)

Abstract: Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration.

## introduction pre-trained language models (plms) @xcite @xcite @xcite built upon transformers @xcite have achieved ground-breaking results across a wide spectrum of language processing tasks such as natural language inference @xcite ), text classification @xcite , named entity recognition @xcite , and part-of-speech tagging @xcite . however, only a few models take into account various linguistic aspects and theories in their design @xcite . morphology and word order of a language are closely related @xcite @xcite ; the latter is incorporated into transformer-based models through positional encoding (pe) @xcite . as language models are being developed for more languages which significantly differ in morphological typology, it could be beneficial to construct language models that are sensitive to these linguistic nuances. moreover, the enormous computational cost incurred during their training is a major challenge in the development of plms. acquiring a deeper understanding of how various components of a plm function in different languages can provide valuable insights regarding their necessity across languages. this motivates us to investigate the relation between positional encoding and morphology, which is essential for wider usage of plms across different languages. our contributions are: 1. performing the first study about the varying importance of positional encoding across languages with different morphological complexity. 2. showing that the impact of pe diminishes as the morphological complexity of a language increases. 3. conducting exhaustive experiments covering 22 different languages across 9 language families and 5 diverse natural language processing tasks. ## related work positional encoding (pe): various methods have been proposed to incorporate position information in transformer models. absolute positions in a sequence, represented by fixed @xcite or trainable encodings @xcite @xcite , are typically added to input embeddings. relative positions are encoded by directly adding position biases into the attention matrix @xcite @xcite @xcite . @xcite introduce rotary positional embeddings, employing a rotation matrix to encode both absolute and relative position information. our study focuses on bert models, which use learnable absolute pes @xcite . we draw insights from linguistics theories and question the design choices for bert-style the child gave father to the king पिता राज्ञे बालक ं दत्तवान् बालक ं राज्ञे पिता दत्तवान् दत्तवान् पिता राज्ञे बालक ं father king_to child_(acc) gave father king_to child_(acc) gave father king_to child_(acc) gave @xcite . english is a morphologically poor language with svo word order whereas sanskrit is a morphologically rich language with no dominant word order (nodom). distorting the word order completely alters the meaning for english. however, for sanskrit the meaning remains intact. models that were designed with english in mind. @xcite , semantics @xcite and morphology @xcite . @xcite al. (2019 ## the relationship between morphology and word order in this section, we investigate the relationship between morphology and word order as outlined in different linguistic theories. linguistic theories : morphological case mark-ings serve a similar function as word order @xcite . several theories suggest that the presence of morphological case is necessary for free word order in a language @xcite . either morphological case or structural position facilitates the unambiguous determination of grammatical role of the constituents of a sentence. the existence of morphological case reduces the need for fixed structural position in determination of the grammatical function of a word or phrase, allowing for variable word order. however, if morphological case is absent, fixed placement of words (and phrases) is necessary, exhibiting a fixed or rigid word order. in this work, we align our empirical study in accordance with the above theories that hints at the existence of a correlation between morphology and word order. specifically, morphologically rich languages which tend to exhibit higher word-order flexibility as compared to morphologically poor languages foot_0 . ## methodology in our work, we first quantify morphological complexity, and then systematically study the effect of removal of positional encodings during fine-tuning. please refer to section 3 for details on linguistic theories governing our study. ## experimental setup to ensure the generalizability of our findings, we choose to perform a comprehensive study spanning different languages and tasks. ## results in this section, we present the findings of our experiments on syntactic and semantic tasks. we also language relative decrease in accuracy (%) 0 2 4 6 8 10 vi zh fr es en sw de el bg ru tr figure 5: effect of positional encoding on xnli. ## conclusion in this work, we demonstrate the interplay between positional encoding and morphology for morphologically diverse languages. we present the first study regarding the varying impact of positional encoding across languages with varying morphological complexity. we cover 22 different languages across 9 language families and 5 diverse natural language processing tasks for our investigation. our results reveal that the importance of positional encoding diminishes as the morphological complexity of a language increases. our study also emphasizes the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration.