A Novel Multi-Task Learning Approach for Context-Sensitive Compound Type Identification in S anskrit (2022)

Abstract: The phenomenon of compounding is ubiquitous in Sanskrit. It serves for achieving brevity in expressing thoughts, while simultaneously enriching the lexical and structural formation of the language. In this work, we focus on the Sanskrit Compound Type Identification (SaCTI) task, where we consider the problem of identifying semantic relations between the components of a compound word. Earlier approaches solely rely on the lexical information obtained from the components and ignore the most crucial contextual and syntactic information useful for SaCTI. However, the SaCTI task is challenging primarily due to the implicitly encoded context-sensitive semantic relation between the compound components. Thus, we propose a novel multi-task learning architecture which incorporates the contextual information and enriches the complementary syntactic information using morphological tagging and dependency parsing as two auxiliary tasks. Experiments on the benchmark datasets for SaCTI show 6.1 points (Accuracy) and 7.7 points (F1-score) absolute gain compared to the state-of-the-art system. Further, our multi-lingual experiments demonstrate the efficacy of the proposed architecture in English and Marathi languages.

## introduction a compound is defined as a collection of one or more entities that act as a single meaningful entity. the process of decoding an implicit semantic relation between the components of a compound in sanskrit is called sanskrit compound type @xcite . alternatively, it is also termed as noun compound interpretation (nci) @xcite . in the literature, the nci problem has been formulated in two related but different ways. let's take mango juice as an example. in the first formulation, the relation between the two components is labeled from a set of semantic relations (madeof) @xcite @xcite . the second formulation uses paraphrasing to illustrate the semantic relations (a juice made from mango) @xcite @xcite . in this work, we use the first formulation that frames the task as a multi-class classification problem. the task is challenging and often depends upon the context or world knowledge about the entities involved @xcite . for instance, the semantic type of the compound rāma-īśvarah . can be classified into one of the following semantic types depending on the context: karmadhāraya foot_1 , bahuvrīhi and tatpurus . a. although the compound has the same components as well as the final form, the implicit relationship between the components can be decoded only with the help of available contextual information @xcite . due to such instances, the downstream natural language processing (nlp) applications for sanskrit such as question answering @xcite and machine translation @xcite , etc. show sub-optimal performance when they stumble on compounds. for example, while translating rāmaīśvarah . into english, depending on the semantic type, there are three possible meanings: (1) lord who is pleasing (in karmadhāraya) (2) the one whose god is rama (in bahuvrīhi) (3) lord of rama (in tatpurus . a). therefore, the sacti task can be seen as a preliminary pre-requisite to building a robust nlp technology for sanskrit. further, this dependency on contextual information rules out the possibility of storing and doing a lookup to identify a compound's semantic types. with the advent of recent contextual models @xcite @xcite , there has been upsurge in performance of various downstream nlp applications @xcite @xcite . nevertheless, there have been no efforts to build context-dependent models in sacti. 3 this may be attributed to the fact that while most of the natural language technology is built for resourcerich languages such as english @xcite , compounding is not a predominant phenomenon in them @xcite . there is also lack of task-specific context-sensitive labeled data. earlier approaches @xcite @xcite for sacti solely rely on lexical information obtained from the components and are blind to potentially useful contextual and syntactic information. the context is the most feasible, cheaply available information. as per pan . ini's grammar (pan . ini, 500 bce; kulkarni and kumar, 2013), the morphological features have direct correlation with the semantic types. sometimes, the dependency information also helps is disambiguation and can provide a medium to enrich contextual information. thus, we propose a novel multi-task learning approach which (1) incorporates the contextual information, and (2) enriches the complementary syntactic information using morphological tagging and dependency parsing auxiliary tasks without any additional manual labeling. summarily, our key contributions are: figure 1 : illustration of the proposed multi-task learning architecture with an example "aham pīta-ambaram namāmi" (translation: "i pray to pītāmbara (lord vishnu).") where 'pīta-ambaram' is a compound word belonging to the bahuvrīhi semantic type as per the given context. we feed this context and the compound word at the end as an input to the system. each token is split into wordpieces using a multi-lingual tokenizer @xcite . this sequence of wordpieces is passed to multi-lingual pretrained xlm-r encoder @xcite . the hidden representation of each token is the average of its wordpieces' representations obtained from the encoder. we apply our multitask learning architecture which consists of three tasks, namely, sanskrit compound type identification (sacti), morphological tagging and dependency parsing over the hidden representations. we formulate sacti as a pairwise (a context word and the compound) classification task where the objective is to predict the semantic type of the target compound word (bahuvrīhi). at test time, we apply the maximum voting policy to select a single prediction from the set of semantic relations predicted by such n pairs. figure 1 illustrates the proposed multi-task learning architecture with an example context, "aham pīta-ambaram namāmi" (translation: "i pray to pītāmbara (lord vishnu).") where 'pīta-ambaram' is a compound belonging to the bahuvrīhi semantic type as per the given context. as shown in figure 1 , we feed this context along with the compound word concatenated at the end, as an input to the system, and obtain hidden representations from the multilingual encoder as described below. on top of the hidden representations as obtained via the encoder module, we apply our multi-task learning architecture consisting of three tasks: sacti, morphological tagging, and dependency parsing. we formulate the sacti task as a pair-wise (a context word paired with the compound word) classification task where the objective is to predict the semantic type of the target compound word (bahuvrīhi for pītaambaram compound word in this example) for all the pairs as shown in figure 1 . at test time, we apply the maximum voting policy to select a single prediction from the set of semantic types predicted by these pairs. we formally discuss the details of each component below. multilingual encoder: sanskrit is a morphologically rich and low-resource language. in order to build powerful contextual representations for sanskrit words, morphological richness poses the outof-vocabulary problem and low-resource nature poses an unlabelled data scarcity problem. thus, we opt for a multi-lingual encoder @xcite to mitigate these issues. given a compound c p and its context @xmath0, c 2 , ..., c n ] such that p th position (1 ≤ @xmath1, we append the compound word to the context such that @xmath2, c 2 , ..., c n , c n+1 ] where c n+1 = c p . each token (c i ) is further split into wordpieces sacti: our context-sensitive classifier uses bi-affine attention, henceforth referred to as biaff. given the hidden representations of i th , j th context words as h i , h j from the multi-lingual encoder, the scoring function s i,j indicates the system's belief that the latter (j th ) (eqn. 2) should be related to the former (i th ) in identifying the semantic type of the latter, where q t i z i indicates bias to capture the prior of contextual information in the i th word. and q ′ k are learnable parameters. finally, model maximizes the following objective function during training. at test time, we apply maximum voting policy to select a single prediction from the set of semantic relations predicted for n pairs morphological tagging: the primary motivation behind using morphological tagging as an auxiliary task aligns well with grammatical rules from pan . ini's grammar (pan . ini, 500 bce; @xcite . for instance, avyayībhāva compounds are in neuter gender. tatpurus . a is a function of 'case' attribute of morphological features. the number attribute of a compound depends on the semantic type of the compound. also, there are constraints based on inflection/derivational suffix. summarily, these morphological features have direct correlation with the semantic classes. in our proposed system, the morphological tagging task leverages hidden representations from the multilingual encoder and decodes the pseudo-labels foot_2 using a fully connected layer followed by a softmax layer. 5 in this process, morphological information useful for the sacti task is enriched in the hidden representations. this can be seen as an implicit way to encode the grammatical rules in the system. dependency parsing: lowe (2015) argues that compounding is mostly a syntactic phenomenon. bahuvrīhi compounds are "exocentric" in nature, which attribute a property to an entity external to the compound with the adjective relationship. sometimes, syntactic information can provide a complementary signal useful for compound type disambiguation. for instance, consider the following example: aham nīla-utpalah . sarah . paśyāmi (translation: i watch the pond having a blue-lotus.) here, nīla-utpalam qualifies to be bahuvrīhi due to presence of its referent sarah . with an adjective relationship. however, in the absence of sarah . in the context, ambiguity pops up in between bahuvrīhi and tatpurus . a. foot_4 this motivates us to investigate the usefulness of syntactic information for the sacti task. the benchmark datasets do not have a gold standard dependency information. we use predicted dependency trees as pseudo-labels ( § 3.1). our dependency parsing component leverages bi-affine parser @xcite over hidden representations from multi-lingual encoder. ## analysis in this section, we dive deep into the proposed system architecture for a detailed analysis as well as generalizability. we use sacti-base coarse dataset in the w/ context setting for the analysis. figure 3 illustrate how the same compound (pītaambaram) in two different contexts [(a-b) aham pīta-ambaram vastram dharāmi (i wear a yellow cloth) and (c-d) aham pīta-ambarām namāmi (i pray to the lord vis . nu)] leads to different semantic type predictions (tatpurus . a: yellow cloth and bahuvrīhi: lord vis . nu). in the dependency heatmap of the first case, pīta-ambaram focuses on vastram (cloth) and in latter case, it focuses on namāmi (the action of praying). as per the grammatical rules, the morphological tagging task correctly predicts the gender information as neuter and masculine in these cases, respectively. thus, this probing analysis suggests that auxiliary tasks not only help add complementary signals to the system but also serve as a proxy for explainability. the results in sanskrit compound type identification task has garnered considerable attention of the researchers in the last decade. in order to decode the meaning of a sanskrit compound, it is essential to figure out its constituents @xcite @xcite , how the constituents are grouped (kulkarni and kumar, 2011), identify the semantic relation between them (kumar, 2012) and finally generate the paraphrase of the compound @xcite . @xcite and @xcite proposed a rule-based approach where around 400 rules mentioned in pān . ini's grammar (pan . ini, 500 bce) were analysed from the perspective of compound generation and type identification, respectively. recently, @xcite investigated whether a purely engineering data-driven approach competes with the performance of a linguistically motivated hybrid approach by @xcite . summarily, no attention has been given to incorporating contextual information, which is crucial and cheaply available. we address this research gap and mark the new state-of-the-art results with substantial improvements. ## conclusion and discussion this work focused on sanskrit compound type identification, where the task is to decode the semantic information hidden in the compound, which can be context-sensitive. this poses a limitation to the existing context agnostic approaches, thus we propose a novel multi-task learning architecture which incorporates the contextual information and also enriches it with complementary syntactic information using morphological tagging and dependency parsing auxiliary tasks. our probing analysis showcased that these auxiliary tasks also serve as a proxy for model prediction explainability. to the best of our knowledge, this is the first time that the importance of these auxiliary tasks has been showcased for sacti. our experiments on benchmark datasets showed that the proposed system provides stunning improvements with 6.1 points (a) and 7.7 points (f1) absolute gain compared with the current stateof-the-art system. our fine-grained analysis showcased some light on the inner engineering of the proposed system. our multi-lingual experiments on english and marathi languages proved the efficacy of the proposed system in other languages. we limit our study to the purely engineering data-driven settings. we plan to extend the current work by augmenting logical rules @xcite derived from pān . inian grammar in the proposed approach.