CSSL : Contrastive Self-Supervised Learning for Dependency Parsing on Relatively Free Word Ordered and Morphologically Rich Low Resource Languages (2024)

Abstract: Neural dependency parsing has achieved remarkable performance for low resource morphologically rich languages. It has also been well-studied that morphologically rich languages exhibit relatively free word order. This prompts a fundamental investigation: Is there a way to enhance dependency parsing performance, making the model robust to word order variations utilizing the relatively free word order nature of morphologically rich languages? In this work, we examine the robustness of graph-based parsing architectures on 7 relatively free word order languages. We focus on scrutinizing essential modifications such as data augmentation and the removal of position encoding required to adapt these architectures accordingly. To this end, we propose a contrastive self-supervised learning method to make the model robust to word order variations. Furthermore, our proposed modification demonstrates a substantial average gain of 3.03/2.95 points in 7 relatively free word order languages, as measured by the UAS/LAS Score metric when compared to the best performing baseline.

## introduction dependency parsing for low-resource languages has greatly benefited from diverse data-driven strategies, including data augmentation @xcite , multi-task learning @xcite , cross-lingual transfer @xcite , self-training @xcite and pre-training @xcite . further, incorporating morphological knowledge substantially improves the parsing performance for low-resource morphologically-rich languages (mrls; @xcite @xcite @xcite . mrls tend to have sentences that follow a relatively free-word-order @xcite , as structural information is often encoded using morphological markers rather than word order. in mrls, a sentence may have different acceptable word order configurations, that preserve the semantic and structural information. however, the permutation invariance is often not reflected in their corresponding semantic space representations encoded using a pretrained model. pretrained models typically include a position encoding component, often shown to be beneficial for tasks in languages that follow a fixed word order. however, removing the position encoding of the encoder during fine-tuning is demonstrated to be counterproductive @xcite . languages, including mrls, tend to follow a preferred word order typology. however, such preferences are often followed for the efficiency of communication, from a cognitive, psycholinguistic, and information-theoretic standpoint and not due to any limitations of the morphology @xcite @xcite . for instance, sanskrit, a classical language @xcite , predominantly consists of sentences written as verses in its pre-classic and classic literature. the majority of the available corpora in sanskrit are written in verse form @xcite . here, verbal cognition often takes a backseat as words are often reordered to satisfy metrical constraints in prosody @xcite . hence, these sentences appear to be arbitrarily ordered based on syntactic analysis @xcite . in this work, we propose a self-supervised contrastive learning framework, primarily for sanskrit, that makes the model agnostic to the word order variations within a sentence. ātmā satyasya jñānena paramam amṛtatvaṁ vindate kartā sambandha karaṇam viśeṣaṇam karma (a) original word order jñānena satyasya amṛtatvaṁ paramam ātmā vindate sambandha karma viśeṣaṇam kartā karaṇam (b) permuted word order of using annotated pairs in contrastive learning @xcite ; @xcite to make the model permutation invariant to the arbitrary word order variations in sanskrit @xcite . given the comprehensive morphological marking system, the core semantic essence of a sentence remains unaltered, rendering the permuted counterpart as a suitable positive pairing for contrastive learning. figure 1 shows that the original sentence and its permuted counterpart having the same dependency structure. this simple use of word permutations in a sentence as positive pairs achieves substantial improvement over prior methods. our approach, to the best of our knowledge, is the first to use a contrastive learning approach for dependency parsing. our proposed approach is modular and agnostic, allowing for seamless integration with any encoder architecture without necessitating alterations to the pretaining decisions. moreover, our objective is to leverage recent advancements in parsing literature by augmenting with the cssl framework, which would make these models more robust to word order variations. in this work, we start by examining the robustness of graph-based parsing architectures @xcite @xcite . we believe, graph-based parsing architectures could be a natural choice to model flexible word order. we then focus on investigating essential modifications such as data augmentation @xcite and the removal of position encoding @xcite required to adapt these architectures accordingly. we finally show the efficacy of our approach on the best baseline @xcite model by integrating cssl with it and report an average performance gain of 3.03/2.95 points (uas/las) improvement over 7 mrls. our main contributions are as follows: the classification loss l ce is the cross-entropy loss applied only to token-level labels of the original training input. the scorer is based on biaffinescorer @xcite , which tries to independently maximize the local probability of the correct head word for each word. ## conclusion in this work, we investigated the robustness of graph-based parsing architectures across 7 languages characterized by relatively flexible word order. we introduced a self-supervised contrastive learning module aimed at making encoders insensitive to variations in word order within sentences. additionally, the modular nature of our approach enables seamless integration with any encoder architecture without necessitating modifications to pretraining decisions. to the best of our knowledge, our approach represents the first utilization of contrastive learning techniques for dependency parsing to address challenges arising from variable word order in low-resource settings. finally, we demonstrate the effectiveness of our approach by integrating it with the rngtr architecture mohammadshahi and henderson (2021), reporting an average performance improvement of 3.03/2.95 points (uas/las) across the 7 mrls. limitations we could not evaluate on complete ud due to limited available compute resources (single gpu); hence, we selected 7 representative languages for our experiments.