S anskrit S hala: A Neural S anskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposes (2023)

Abstract: We present a neural Sanskrit Natural Language Processing (NLP) toolkit named SanskritShala (a school of Sanskrit) to facilitate computational linguistic analyses for several tasks such as word segmentation, morphological tagging, dependency parsing, and compound type identification. Our systems currently report state-of-the-art performance on available benchmark datasets for all tasks. SanskritShala is deployed as a web-based application, which allows a user to get real-time analysis for the given input. It is built with easy-to-use interactive data annotation features that allow annotators to correct the system predictions when it makes mistakes. We publicly release the source codes of the 4 modules included in the toolkit, 7 word embedding models that have been trained on publicly available Sanskrit corpora and multiple annotated datasets such as word similarity, relatedness, categorization, analogy prediction to assess intrinsic properties of word embeddings. So far as we know, this is the first neural-based Sanskrit NLP toolkit that has a web-based interface and a number of NLP modules. We are sure that the people who are willing to work with Sanskrit will find it useful for pedagogical and annotative purposes. SanskritShala is available at: https://cnerg.iitkgp.ac.in/sanskritshala . The demo video of our platform can be accessed at: https://youtu.be/x0X31Y9k0mw4 .

## introduction sanskrit is a culture-bearing and knowledgepreserving language of ancient india. digitization has come a long way, making it easy for people to access ancient sanskrit manuscripts @xcite . however, we find that the utility of these digitized manuscripts is limited due to the user's lack of language expertise and various linguistic phenomena exhibited by the language. this motivates us to investigate how we can utilize natural language technologies to make sanskrit texts more accessible. the aim of this research is to create neural-based sanskrit nlp systems that are accessible through a user-friendly web interface. the sanskrit language presents a range of challenges for building deep learning solutions, such as the sandhi phenomenon, a rich morphology, frequent compounding, flexible word order, and limited resources @xcite @xcite . to overcome these challenges, 4 preliminary tasks were identified as essential for processing sanskrit texts: word segmentation, morphological tagging, dependency parsing, and compound type identification. the word segmentation task is complicated by the sandhi phenomenon, which transforms the word boundaries @xcite . the lack of robust morphological analyzers makes it challenging to extract morphological information, which is crucial for dependency parsing. similarly, dependency information is essential for several downstream tasks such as word order linearisation @xcite which helps to decode possible interpretation of the poetic composition. additionally, the ubiquitous nature of compounding in sanskrit is difficult due to the implicitly encoded semantic relationship between its constituents @xcite . these 4 tasks can be viewed as a preliminary requirement for developing robust nlp technology for sanskrit. thus, we develop novel neural-based linguistically informed architectures for all 4 tasks, reporting state-of-theart performance on sanskrit benchmark datasets @xcite . in this work, we introduce a neural sanskrit nlp toolkit named sanskritshala 2 to assist computational linguistic analyses involving multiple tasks such as word segmentation, morphological tagging, dependency parsing, and compound type identification. sanskritshala is also deployed as a web application that enables users to input text and gain real-time linguistic analysis from our pretrained systems. it is also equipped with user-friendly interactive data annotation capabilities that allow annotators to rectify the system when it makes errors. it provides the following benefits: (1) a user with no prior experience with deep learning can utilise it for educational purposes. (2) it can function as a semi-supervised annotation tool that requires human oversight for erroneous corrections. we publicly release the source code of the 4 modules included in the toolkit, 7 word embedding models that have been trained on publicly available sanskrit corpora and multiple annotated datasets such as word similarity, relatedness, categorization, analogy prediction to measure the word embeddings' quality. to the best of our knowledge, this is the first neural-based sanskrit nlp toolkit that contains a variety of nlp modules integrated with a web-based interface. summarily, our key contributions are as follows: ## related work on sanskrit nlp tools recently, the sanskrit computational linguistics (scl) field has seen significant growth in building web-based tools to help understand sanskrit texts. @xcite introduced the sanskrit heritage reader (shr), a lexicon-driven shallow parser that aids in the selection of segmentation solutions. samsādhanī is another web-based tool consisting of various rule-based modules @xcite @xcite . recently, @xcite introduced a web-based annotation tool for knowledge-graph construction and a metrical analysis. in short, tools for nlp can be divided into two groups: rule-based and annotation tools. rulebased tools have limitations such as not providing a final solution, limited vocabulary coverage, and lacking user-friendly annotation features. annotation tools, on the other hand, do not have the recommendations of rule-based systems, relying solely on annotators. to address these limitations, a web-based annotation framework called shr++ @xcite was proposed. it combines the strengths of both types of tools by offering all possible solutions from rule-based system shr for tasks like word segmentation and morphological tagging, allowing annotators to choose the best solution rather than starting from scratch. our proposal, sanskritshala, goes a step further by integrating a neural-based nlp toolkit that combines state-of-the-art neural-based pre-trained models with rule-based suggestions through a webbased interface. each module of sanskritshala is trained to predict the solutions from the exhaustive candidate solution space generated by rule-based systems. hence, it makes predictions in real time using neural-based models that have already been trained. thus, a complete solution is shown to the users / annotators, which was not possible in any of the previous attempts. further, annotators can easily correct the mispredictions of the system with the help of user-friendly web-based interface. this would significantly reduce the overall cognitive load of the annotators. to the best of our knowledge, sanskritshala is the first nlp toolkit available for a range of tasks with a user friendly annotation interface integrated with the neural-based modules. ## about sanskrit sanskrit is an ancient language known for its cultural significance and knowledge preservation. however, it presents challenges for deep learning due to its morphological complexity, compounding, free word order, and lack of resources. sanskrit's intricate grammar, with its combination of roots, prefixes, and suffixes, requires advanced algorithms to analyze and understand. compounding adds another layer of complexity as multiple words combine to form new words with unique meanings @xcite . the free word order in sanskrit complicates tasks transformer encoder with sma module ... u dāsaḥ dāsa bhava s ... 4 1 1 5 3 … 4 4 3 8 3 ... b s ā 2 2 ā d 1 1 d o 4 4 aḥ b 5 5 (a) char embed word embed t 1 t 11 t 1r t k t k1 t kr loung attention on character t i o 1 o k e i,1 e i,2 e i,m1 e i o i e i l i,1 l i,2 l i,n1 2. morph tag prediction head 3. lemma prediction head 1. char + word encoder e i,1 e i,2 ## a neural nlp sanskrit toolkit in this section, we describe sanskritshala, which is a neural sanskrit nlp toolkit designed to aid computational linguistic analysis including various tasks, such as word segmentation, morphological tagging, dependency parsing, and compound type identification. it is also available as a web application that allows users to input text and obtain real-time linguistic analysis from our pretrained algorithms. we elucidate sanskritshala by first elaborating on its key modules. word tokenizer: earlier lexicon-driven systems for sanskrit word segmentation (sws) rely on sanskrit heritage reader @xcite , a rule-based system, to obtain the exhaustive solution space for segmentation, followed by diverse approaches to find the most valid solution. however, these systems are rendered moot while stumbling out-of-vocabulary words. later, data-driven systems for sws are built using the most recent techniques in deep learning, but can not utilize the available candidate solution space. to overcome the drawbacks of both lines of modelling, we build a transformerbased linguistically-informed sanskrit tokenizer (translist) @xcite containing (1) a component that encodes the character-level and word-level potential candidate solutions, which tackles sandhi scenario typical to sws and is compatible with partially available candidate solution space, (2) a novel soft-masked attention for prioritizing selected set of candidates and (3) a novel path ranking module to correct the mispredictions. figure 1 (a) illustrates the translist architecture, where the candidate solutions obtained from shr are used as auxiliary information. in terms of the perfect match (pm) metric, translist outperforms with 93.97 pm compared to the state-of-the-art @xcite with 87.08 pm. ## sanskrit resources in sanskritshala in this section, we describe 7 word embeddings pretrained on sanskrit corpora and suit of 4 intrinsic tasks datasets to assess the quality of word embeddings, followed by the description of web interface. ## conclusion we present the first neural-based sanskrit nlp toolkit, sanskritshala which facilitates diverse linguistic analysis for tasks such as word segmentation, morphological tagging, dependency parsing and compound type identification. it is set up as a web-based application to make the toolkit easier to use for teaching and annotating. all the codebase, datasets and web-based applications are publicly available. we also release word embedding models trained on publicly available sanskrit corpora and various annotated datasets for 4 intrinsic evaluation tasks to assess the intrinsic properties of word embeddings. we strongly believe that our toolkit will benefit people who are willing to work with sanskrit and will eventually accelerate the sanskrit nlp research.