S ans GPT : Advancing Generative Pre-Training in S anskrit (2024)

Abstract: In the past decade, significant progress has been made in digitizing Sanskrit texts and advancing computational analysis of the language. However, efforts to advance NLP for complex semantic downstream tasks like Semantic Analogy Prediction, Named Entity Recognition, and others remain limited. This gap is mainly due to the absence of a robust, pre-trained Sanskrit model built on large-scale Sanskrit text data since this demands considerable computational resources and data preparation. In this paper, we introduce SansGPT, a generative pre-trained model that has been trained on a large corpus of Sanskrit texts and is designed to facilitate fine-tuning and development for downstream NLP tasks. We aim for this model to serve as a catalyst for advancing NLP research in Sanskrit. Additionally, we developed a custom tokenizer specifically optimized for Sanskrit text, enabling effective tokenization of compound words and making it better suited for generative tasks. Our data collection and cleaning process encompassed a wide array of available Sanskrit literature, ensuring comprehensive representation for training. We further demonstrate the model’s efficacy by fine-tuning it on Semantic Analogy Prediction and Simile Element Extraction, achieving an impressive accuracy of approximately 95.8% and 92.8%, respectively.

## introduction sanskrit, one of the most ancient and highly structured natural languages, consists of a vast and 1 by conservative estimate the time period of the oldest text in sanskrit i.e., ṛgveda is considered as 1000 b.c. diverse corpus of literature, with domains spanning from foundational texts on vyākaraṇa (grammar), śikṣā (phonetics), and nirukta (etymology), to critical works on mīmāmsā (exegesis), nyāya (logic), kāvyaśāstra (poetics), sāhitya (literature), nāṭyaśāstra (dramaturgy), dharmaśāstra (jurisprudence), etc. with origins tracing back to approximately 1000 bce 1 , sanskrit remains in use today, not only as a language of traditional knowledge but also as a subject of modern linguistic and computational studies. in recent years, significant efforts have been directed towards the digitization of sanskrit texts and the advancement of natural language processing (nlp) techniques tailored to process this classical language. notable progress has been made in tasks such as word segmentation @xcite , dependency parsing @xcite , and word-order linearization @xcite . these advancements have been pivotal in addressing some of the unique challenges posed by sanskrit's intricate grammatical and syntactic structures. despite these strides, there remains a notable gap in developing nlp models capable of handling more complex semantic tasks such as semantic analogy prediction, named entity recognition, and other sophisticated downstream applications. the lack of a robust sanskrit model trained on diverse, large-scale data is a significant barrier to advancing complex nlp tasks in the language. models pre-trained on extensive datasets are critical for evaluating downstream tasks. to address this, we introduce sansgpt, a pre-trained sanskrit language model aimed at advancing nlp capabilities, particularly for semantic tasks, thereby simplifying the research and development of downstream tasks. our contributions are: 1) release of a cleaned pre-training sanskrit corpus foot_0 : we provide a large, meticulously cleaned, and pre-processed sanskrit corpus consisting of approximately 51 million tokens. this corpus has been cleaned and prepared specifically for use in gpt-based pre-training. 2) development of a custom sanskrit tokenizer: we developed a lightweight and efficient tokenizer using byte pair encoding tailored for sanskrit. it breaks long compound words while preserving the syntactic structure (e.g., sandhi) for coherent text generation, ensuring that the tokenization maintains word integrity and natural flow. 3) pre-training and release 2 of sansgpt: we pre-train and release sansgpt, a generative model designed for the sanskrit language. this model sets the foundation for advancements in generative pre-training and fine-tuning within the sanskrit nlp domain which would help to advance the development of complex semantic down-stream tasks. 4) high performance in semantic analogy prediction, simile element extraction, and release 2 of simile (upamā) element extraction dataset: demonstrating the efficacy of sansgpt, we achieved a remarkable validation accuracy of approximately 95.8% on the semantic analogy prediction task. we also evaluate and release the simile (upamā) element extraction dataset, achieving a validation accuracy of 92.8% on the task. ## evaluation in this section, we evaluate sansgpt's performance on two tasks. we design the tasks, fine-tune the pre-trained model checkpoint, and report the results obtained. ## conclusion and future scope sansgpt represents a significant step in the field of sanskrit natural language processing. by providing a robust, pre-trained model, it addresses the critical gap in the development of advanced tools for analysing and understanding sanskrit texts. the model's ability to effectively handle complex semantic tasks, as demonstrated by its evaluation performance, highlights its potential to facilitate a wide range of applications. sansgpt offers a solid starting point for various downstream tasks, including named entity recognition, text summarization, and sentiment analysis.