N-gram-Based Preprocessing for Sandhi Reversion in V edic S anskrit (2024)

Abstract: This study aims to address the challenges posed by sandhi in Vedic Sanskrit, a phenomenon that complicates the computational analysis of Sanskrit texts. By focusing on sandhi reversion, the research seeks to improve the accuracy of processing Vedic Sanskrit, an older layer of the language. Sandhi, a phonological phenomenon, poses challenges for text processing in Sanskrit due to the fusion of word boundaries or the sound change around word boundaries. In this research, we developed a transformer-based model with a novel n-gram preprocessing strategy to improve the accuracy of sandhi reversion for Vedic. We created character-based n-gram texts of varying lengths (n = 2, 3, 4, 5, 6) from the Rigveda, the oldest Vedic text, and trained models on these texts to perform machine translation from post-sandhi to pre-sandhi forms. In the results, we found that the model trained with 5-gram text achieved the highest accuracy. This success is likely due to the 5-gram’s ability to capture the maximum phonemic context in which Vedic sandhi occurs, making it more effective for the task. These findings suggest that by leveraging the inherent characteristics of phonological changes in language, even simple preprocessing methods like n-gram segmentation can significantly improve the accuracy of complex linguistic tasks.

## introduction 1.sandhi in sanskrit vedic sanskrit represents an earlier stage of the sanskrit language, characterized notably by its unique system of accentuation, where the meaning of words in vedic can vary depending on the position of the accent. sandhi refers to a phonological/phonetic phenomenon wherein certain conditions induce changes in the sounds of adjacent words or morphemes @xcite . this phenomenon is divided into two types: external sandhi, which involves sound changes at the junctions between words in a sequence, and internal sandhi, which affects the internal structure of words, such as in the derivation of nouns and verbs. for instance, when the words yás ("who/which", a relative pronoun in the nominative singular masculine), hatvā́("having killed", absolutive of the verb han), and áhim ("a snake", a noun in the accusative singular) consecutively in a sentence, they have a surface form like yó hatvā́him (rigveda 2.12.3a) meaning "[indra,] who [...] after killing the snake". the final two sounds of yás change to -ó before the voiced sound h-. additionally, the final vowel -ā́of hatvā́and the initial vowel áof áhim merge into -ā-. conversely, when the same word hatvā́is followed by pr̥ thivyā́m ("on the earth"), it becomes hatvā́pr̥ thivyā́m (rigveda 1.100.18b), meaning "after [he] struck [dasyus and śimyus, felled them] on the earth." in this case, the final vowel -ā́of hatvā́does not alter before the voiceless stop p-. hence, external sandhi varies according to the boundary sounds of adjacent words in a sentence. internal sandhi, in contrast, can alter the sound at the morpheme boundary, as illustrated by the example of the genitive singular form of the stem tanú-("body") is tanv-às. internal sandhi also undergoes variations depending on the adjacent sounds, similar to external sandhi. in general, sanskrit texts preserve the forms resulting from sandhi rules as they are. therefore, to analyze sanskrit texts, it is essential to identify the original word forms prior to the sound changes imposed by sandhi. despite the precise definition of sandhi rules, reversing the transformations to recover pre-sandhi forms poses significant challenges, whether done manually or via computational methods. to accurately revert hatvā́him back to the two separate words hatvā́and áhim, one must have knowledge of the word forms: the absolutive hatvā́of the verb han ("kill") and the accusative singular áhim of the noun áhi-("snake"). when attempting to segment the phonetic sequence h-a-t-v-ā-h-i-m into two parts, one must first determine the necessity of splitting. if splitting is required, a sequence containing a long vowel -āwithout consideration of accent, may yield four potential outcomes: -ă ă-, -ă ā-, -ā ă-, and -ā ā-(for clarity, short vowels are indicated with ă.). furthermore, when taking accent into account, numerous additional possibilities arise, including sequences where both parts bear an accent, or sequences where only one part is accented, e.g. ihā́@xmath0, índrā́= índra ā́. ## related work given the vast corpus of sanskrit texts, manually reverting all instances of sandhi is impractical and time-consuming. as a result, numerous computational methods for sandhi reversion have been developed over the years to automate this process. recent research has shown that machine learning-based sandhi reversion is more accurate than previous rule-based approaches. the following sections review major studies that have employed machine learning techniques for sandhi reversion. ## method this study addresses the task of translating text from post-sandhi to pre-sandhi forms, building upon the approaches discussed in the related work section (section 2). the objective is to develop a model that accurately reverses the sandhi processes, thereby restoring the original word boundaries in vedic sanskrit texts. ## results for comparative purposes, the training and evaluation times for each model are not a primary focus of this study, because the computational efficiency is less of a concern for sanskrit, an ancient language with a relatively fixed corpus, compared to the modern languages with increasing linguistic resources. the results indicate that the sandhi reversion model trained on 5-gram text achieves the highest precision, recall, and f1 scores among all the models tested. notably, these scores are slightly higher than those reported for the most accurate sandhi reversion model using the combined rnn + cnn approach by @xcite , even though the datasets used in the two studies differ. ## limitations while the proposed n-gram-based preprocessing method showed promising results, several limitations remain. first, the dataset used in this study was limited to the rigveda, and the model's effectiveness on other vedic or classical sanskrit texts remains to be evaluated. additionally, although the 5-gram model performed well, this approach may not generalize to texts with different phonological structures, such as accentless sanskrit texts. another limitation is the reliance on the padapāṭha as the pre-sandhi form, which is one of several possible interpretations of the rigveda. ## conclusion the results of this study demonstrate that some models utilizing n-gram preprocessing outperform the model with no special text processing. the highest accuracy is particularly achieved when @xmath2. this n value is thought to be related to the phonological environment in which sandhi occurs. the maximum number of phonemes in the environment where sandhi occurs is five: /a s + x -; where / denotes an accent, + denotes a space, and x represents an arbitrary voiced sound. therefore, when @xmath3, the n-gram text adequately includes this sequence, potentially resulting in the highest accuracy. however, it is predicted that @xmath4, which does not have accents. by demonstrating the effectiveness of n-gram preprocessing, this study shows that the approach is independent of the specific translation architecture used. this flexibility implies that our method can be adapted to future innovations in translation tasks, regardless of the underlying architecture. although the corpus of vedic texts is smaller than that of classical texts, vedic language processing remains in its early stages. implementing sandhi reversion as a preprocessing technique significantly advances vedic nlp by facilitating tasks such as named entity recognition and intertextual comparison based on word occurrences. furthermore, this n-gram segmentation method is not restricted to the sandhi reversion problem in vedic sanskrit; it is also applicable to word segmentation challenges in classical sanskrit, which lacks accents, as well as in other languages. in these contexts, using n-grams with a size close to or matching the maximum string length of the relevant phonetic sequences in each language is expected to yield high accuracy.