T rans LIST : A Transformer-Based Linguistically Informed S anskrit Tokenizer (2022)

Abstract: Sanskrit Word Segmentation (SWS) is essential in making digitized texts available and in deploying downstream tasks. It is, however, non-trivial because of the sandhi phenomenon that modifies the characters at the word boundaries, and needs special treatment. Existing lexicon driven approaches for SWS make use of Sanskrit Heritage Reader, a lexicon-driven shallow parser, to generate the complete candidate solution space, over which various methods are applied to produce the most valid solution. However, these approaches fail while encountering out-of-vocabulary tokens. On the other hand, purely engineering methods for SWS have made use of recent advances in deep learning, but cannot make use of the latent word information on availability. To mitigate the shortcomings of both families of approaches, we propose Transformer based Linguistically Informed Sanskrit Tokenizer (TransLIST) consisting of (1) a module that encodes the character input along with latent-word information, which takes into account the sandhi phenomenon specific to SWS and is apt to work with partial or no candidate solutions, (2) a novel soft-masked attention to prioritize potential candidate words and (3) a novel path ranking algorithm to rectify the corrupted predictions. Experiments on the benchmark datasets for SWS show that TransLIST outperforms the current state-of-the-art system by an average 7.2 points absolute gain in terms of perfect match (PM) metric.

## introduction sanskrit is considered as a cultural heritage and knowledge preserving language of ancient india. the momentous development in digitization efforts has made ancient manuscripts in sanskrit readily available for the public domain. however, the usability of these digitized manuscripts is limited due to linguistic challenges posed by the language. sws conventionally serves the most fundamental prerequisite for text processing step to make these digitized manuscripts accessible and to deploy many downstream tasks such as text classification @xcite , morphological tagging @xcite , dependency parsing @xcite , automatic speech recognition @xcite etc. sws is not straightforward due to the phenomenon of sandhi, which creates phonetic transformations at word boundaries. this not only obscures the word boundaries but also modifies the characters at juncture point by deletion, insertion and substitution operation. figure 1 illustrates some of the syntactically possible splits due to the language-specific sandhi phenomenon for sanskrit. this demonstrates the challenges involved in identifying the location of the split and the kind of transformation performed at word boundaries. the recent surge in sws datasets @xcite has led to various methodologies to handle sws. existing lexicondriven approaches rely on a lexicon driven shallow parser, popularly known as sanskrit heritage reader (shr) @xcite ). 2 this line of approaches @xcite @xcite formulate the task as finding the most accurate semantically and syntactically valid solution from the candidate solutions generated by shr. with the help of the significantly reduced exponential search space provided by shr and linguistically involved feature engineering, these lexicon driven systems @xcite report close to stateof-the-art performance for the sws task. however, these approaches rely on the completeness assumption of shr, which is optimistic given that shr does not use domain specific lexicons. these models are handicapped by the failure of this preliminary step. on the other hand, purely engineering based knowledge-lean data-centric approaches @xcite @xcite perform surprisingly well without any explicit hand-crafted features and external linguistic resources. these purely engineering based approaches are known for their ease of scalability and deployment for training/inference. however, a drawback of these approaches is that they are blind to latent word information available through external resources. there are also lattice-structured approaches @xcite @xcite (originally proposed for chinese named entity recognition (ner), which incorporate lexical information in character-level sequence labelling architecture). however, these approaches cannot be directly applied for sws; since acquiring word-level information is not trivial due to sandhi phenomenon. to overcome these shortcomings, we propose transformer-based linguistically informed tokenizer (translist). translist is a perfect blend of purely engineering and lexicon driven approaches for the sws task and provides the following advantages: (1) similar to purely engineering approaches, it facilitates ease of scalability and deployment during training/inference. our key contributions are as follows: (a) we propose the linguistically informed tokenization module ( § 2.1) which accommodates language-specific sandhi phenomenon and adds inductive bias for the sws task. (b) we propose a novel soft-masked attention ( § 2.2) that helps to add inductive bias for prioritizing potential candidates keeping mutual interactions between all candidates intact. (c) we propose a novel path ranking algorithm ( § 2.3) to rectify the corrupted predictions. (d) we report an average 7.2 points perfect match absolute gain ( § 3) over the current state-of-the-art system @xcite . we elucidate our findings by first describing translist and its key components ( § 2), followed by the evaluation of translist against strong baselines on a test-bed of 2 benchmark datasets for the sws task ( § 3). finally, we investigate and delve deeper into the capabilities of the proposed components and its corresponding modules ( § 4). ## methodology in this section, we will examine the key components of translist which includes a linguistically informed tokenization module that encodes character input with latent-word information while accounting for sws-specific sandhi phenomena ( § 2.1), a novel soft-masked attention to prioritise potential candidate words ( § 2.2) and a novel path ranking algorithm to correct mispredictions ( § 2.3). ## analysis in this section, we investigate various questions to dive deeper into the proposed components and investigate the capabilities of various modules. we use sighum dataset for the analysis. for example, the potential outputs for the input ā can be ā, ā-ā, ā-a, a-a and ah . . the correct rule can be decided based on the context. these multiple rules pose a non-trivial challenge for a system to identify the applicability of specific rule. therefore, it is interesting to compare the translist with current state-of-the-art system to verify its ability for semantic generalization. we observe that translist consistently outperforms rcnn-ss in terms of all metrics. 9 ## related work earlier approaches on sws focused on rule-based finite state transducer systems @xcite . @xcite attempted to solve the sws task for sentences with one or two splits using the bayesian approach. recently, goyal and huet (2016a, shr) proposed a lexicon driven shallow parser. this, along with the recent upsurge in segmentation datasets @xcite @xcite led to two categories of approaches, namely, lexicon driven @xcite @xcite and purely engineering @xcite @xcite . these existing approaches for sws are either brittle in realistic scenarios or do not consider the potentially useful/available information. thus, translist bridges the shortcomings exhibited by each family and gives a win-win solution that marks a new state-of-the-art results. ## conclusion and discussion in this work, we focused on sanskrit word segmentation task. to address the shortcomings of existing purely engineering and lexicon driven approaches, we demonstrate the efficacy of translist as a winwin solution over drawbacks of the individual lines of approaches. translist induces inductive bias for tokenization in a character input sequence using the list module, and prioritizes the relevant candidate words with the help of soft-masked attention (sma module). further, we propose a novel path ranking algorithm to rectify corrupted predictions using linguistic resources on availability (prcp module). our experiments showed that translist provides a significant boost with an average 7.2 points (pm) absolute gain compared to the best baselines, rcnn-ss (sighum) and flat-lattice (hackathon). we have also showcased fine-grained analysis on translist's inner working. we plan to extend this work for morphological tagging in standalone mode @xcite and multi-task setting @xcite with the sws task.