S anskrit Segmentation revisited (2019)

Abstract: Computationally analyzing Sanskrit texts requires proper segmentation in the initial stages. There have been various tools developed for Sanskrit text segmentation. Of these, Gérard Huet’s Reader in the Sanskrit Heritage Engine analyzes the input text and segments it based on the word parameters - phases like iic, ifc, Pr, Subst, etc., and sandhi (or transition) that takes place at the end of a word with the initial part of the next word. And it enlists all the possible solutions differentiating them with the help of the phases. The phases and their analyses have their use in the domain of sentential parsers. In segmentation, though, they are not used beyond deciding whether the words formed with the phases are morphologically valid. This paper tries to modify the above segmenter by ignoring the phase details (except for a few cases), and also proposes a probability function to prioritize the list of solutions to bring up the most valid solutions at the top.

## introduction every sanskrit sentence in the saṃhitā form (continuous sandhied text) is required to be segmented into proper morphologically acceptable words and the obtained result should agree with syntactic and semantic correctness for it's proper understanding. the obtained segmented text consists of individual words where even the compounds are segmented into their components. and there can be more than one segmentation for the same saṃhitā text. the segmented form does not provide any difference in the sense of the text when compared with the saṃhitā form except for the difference in the phonology of the words where it can be observed that the end part of the initial word together with the first letter of the next word undergoes phonetic change. the saṃhitā form, in fact, represents the text similar to a speech text because the knowledge transfer, in the olden days, was predominantly based on oral rendition. but now, for extracting information from these texts it is necessary that they be broken down into pieces so that the intention of the text is revealed completely without any ambiguity. in order to understand any sanskrit text, this process of breaking down into individual words is necessary, and it is popularly known as sandhi-viccheda (splitting of the joint text) in sanskrit. this process takes into account the morphological analyses of each of the split-parts obtained. as there is always a possibility for multiple morphological analyses even for individual words, considering only the morphological validation might result in enormous number of solutions for long sentences. so, syntactical accuracy is also measured to reduce the number of solutions. even then, there is always a possibility for multiple solutions to remain, which cannot be resolved further without the semantic and contextual understanding of the sentence @xcite . owing to this, we find that there is non-determinism right at the start of linguistic analysis @xcite , since sandhi splitting is the first step in the analysis of a sanskrit sentence. this kind of non-determinism is also found in languages like chinese and japanese, where the word boundaries are not indicated, and also in agglutinative languages like turkish @xcite . in some of these languages like thai @xcite , most of the sentences have mere concatenation of words. possible boundaries are predicted using the syllable information, and the process of segmentation starts with segmenting the syllable first, followed by the actual word segmentation. for chinese though, their characters called hanzi are easily identifiable, and the segmentation could be done by tagging @xcite , or determining the word-internal positions using machine learning or deep learning algorithms (like what is done in @xcite ). in the case of vietnamese @xcite , compound words are predominantly formed by semantic composition from 7000 syllables, which can also exist independently as separate words. this is similar to what can be observed in aluk samāsa in sanskrit, which are rare in occurrence. for languages like english, french and spanish where the boundaries are specifically observed as delimiters like space, comma, semi-colon, full stop, etc., segmentation is done using these delimiters and is comparatively simple. in all the above cases, we find that either there are delimiters to separate the words, or individual words are joined by concatenation which ultimately rests the segmentation process in the identification of boundaries. in the case of sanskrit though, these kinds of words form a very small percentage. rather, there is the euphony transformation that takes place at every word boundary. this transition can be generally stated as u|@xmath0, where u is the final part of the first word, v the first part of the next word, and w the resultant form after combining u and v. here the parts may contain at the most two phonemes. the resultant w may contain additional phonemes or may have elisions, but never are more than two phonemes introduced. so this transition or sandhi (external) occurs only at the phoneme level, and it does not require any other information regarding the individual words used. 1 but the reverse process of segmentation does require a morphological analyzer to validate the segments in a split. 1 in the case of internal sandhi between preverbs and verbs, the lexical knowledge of the preverb is required. and in some compounds (like those denoting a saṃjñā), certain cases of retroflexion is permitted. but in this paper only the external sandhi is considered. and it is entirely up to the speaker or writer to perform these transitions or keep the words separated (called vivakṣā -speaker's intention or desire). but in most of the texts and manuscripts, the sandhi is done throughout the text. so, finding the split location alone will not be enough to segment the texts properly. having looked into some of the intricacies of sandhi in sanskrit, we can come up with a mechanical segmentation algorithm that splits a given text into all possible segments: although this mechanical process looks quite simple, the previously mentioned issues like non-determinism do prevail. and systems like the sanskrit reader in sanskrit heritage engine come up with better ways to try to account these problems. the current paper tries to update these efforts. it is organized as follows: section 2 gives the update on how the segmentation for sanskrit has been dealt with in recent years. section 3 discusses the important features of the sanskrit heritage engine's reader. section 4 explains in detail the issues present in the reader. the modifications needed to be done, and the implementation for this paper compose section 5. it also quotes theoretically the reasons for these modifications and provides the probability function proposed in this paper. section 6 describes the methodology of the implementation, and the results and observations are in section 7. ## current methods achieving the correct segmentation computationally is as much difficult as it is manually. a general approach would be the conversion of the mechanical sandhi splitting process mentioned in section 1 to a working algorithm, followed by checking the statistics available for the frequencies of the words and transitions. but there has been a lot of better research work, both rule-based and statistical, on computational sandhi splitting in sanskrit. @xcite , as a part of the sanskrit heritage engine, developed a segmenter for sanskrit texts using a finite state transducer. two different segmenters were developed -one for internal sandhi, which is deployed in the morphological analyser, and the other for external sandhi. the current paper focuses on updating this external sandhi segmenter. @xcite had used the optimality theory to derive a probabilistic method, and de-veloped two methods to segment the input text (1) by augmenting the finite state transducer developed using openfst @xcite , with sandhi rules where the fst is used for the analysis of the morphology and is traversed for the segmentation, and (2) used optimality theory to validate all the possible segmentations. @xcite developed a compound processor where the segmentation for the compound words was done and used optimality theory with a different probabilistic method (discussed in section 5). @xcite later modified the posterior probability function and also developed an algorithm based on bayesian word segmentation methods with both unsupervised and supervised algorithms. @xcite proposed an approach combining the morphological features and word co-occurrence features from a manually tagged corpus from @xcite , and took the segmentation problem as a query expansion problem and used path constrained random walk framework for selecting the nodes of the graph built with possible solutions from the input. @xcite built a word segmenter that uses a deep sequence to sequence model with attention to predict the correct solution. this is the state of art segmenter with precision and recall as 90.77 and 90.3, respectively. ibm research team @xcite , had built a double decoder rnn with attention as seq2(seq) 2 , where they have emphasized finding the locations of the splits first, and then the finding of the split words. and they have the accuracy as 95% and 79.5% for finding the location of splits and the split sentence, respectively. hellwig and nehrdich (2018 ## heritage segmenter the sanskrit heritage engine's segmenter was chosen for further development, for three reasons -1. it is the best segmenter available online with source code available under gpl. it analyses the given input and produces the split based on three main factors: let us consider the sentence rāmālayo ′ sti, as an example to understand these factors. it can be observed that there are twelve possible split solutions given in other possible words like rā, mālayaḥ, etc. are not taken as proper splits because they do the phase details like iic for rāma or pr for asti, etc. are displayed along with the words. these assignments of the phase information to the words and their analysis are the jobs of the phase feasibility module. to understand these phases, look at figure 3 (the first solution for the sentence rāmālayo ′ sti). rāma is the first split and has the phase iic. ālayaḥ is the second split with two morphological possibilities -ālaya and āli. and the transition between the first two words is -a | ā → ā. figure 3: the first solution for the sentence rāmālayo ′ sti the third split is asti with root as and phase pr. and the transition follows the equationaḥ | @xmath1. these transitions are taken care of by the transition feasibility module and the phases mentioned above are taken care of by the phase feasibility module. according to @xcite , sentences are formed by the image of the relation r (sandhi rules) on kleene closure of w*, of a regular set w of words (vocabulary of inflected words). the sanskrit heritage reader accepts a candidate sentence w, and applies the inverted form of the relation r, thus producing a set of words -w1, w2, w3,.... and each of the individual words are valid according to the rules of morphology, and their combination makes some sense. the methodology followed in the segmenter proposed in @xcite starts with using the finite state transducer for generating the chunks, instead of the traditional recursive method over the sentence employed in many sandhi splitting tools. the fst considers the phases as important characteristics of the words. these phases correspond to a finite set of forms. to understand how a word is obtained, let us first take a small example of how the substantival forms (subantas) are obtained. a subanta is analysed as a nominal stem followed by a suffix. the nominal stem can be either an underived stem or a derived stem. in case of a derived stem, the derivation of this stem is also provided by the segmenter. a compound, for example, has a derived stem which contains a sequence of components followed by a nominal suffix. and three phases are present to represent the subantas: ## issues in heritage segmenter the segmenter is embedded in the sanskrit reader which displays all the outputs with the corresponding split word, it's phase and the transition involved with the subsequent word, except when the number of outputs is huge, in which case it shows only the summary. the reader shows the distinction between words and phases based on verb, noun, iic, inde, etc, but not between some of the case-markers. so, there is inconsistency in disambiguation: sometimes the phase is used for pruning out certain solutions, but in some cases, it is not. for example, rāmovanaṅgacchati produces the following 4 solutions the segmenter provides segmentation and also does partial disambiguation. for example, rāmaḥ is ambiguous morphologically and the machine has correctly disambiguated the alternatives. we see the noun analysis of it in the first and second solutions, and the verbal analysis in third and fourth solutions. but we notice that the word vanam which is ambiguous between two morphological analyses, one with nominative case marker and the other with accusative marker, is not disambiguated. @xcite mention that the consideration of a word's similar declensions as different might result in more ambiguity, and the purpose of the segmentation is to find the morphologically apt words and hence they are taken as one. if we look at these four solutions, at the word level, all of them correspond to rā-maḥ vanam gacchati. in order to decide the correct solution among the four, we need to do syntactico-semantic analyses that depend solely upon the linguistic or grammatical information in the sentence @xcite . ## proposed modification we notice that the use of phase information results in multiple solutions. in order to choose the correct solution among them, one needs to look beyond the word analysis and look at the possible relations between the words. this is the domain of the sentential parser. only a sentential parser can decide which of the segmentations with phase information is the correct one. thus we do not see any advantage of having the phase information. and in the interface, the system is not uniform in resolving the ambiguities. it uses certain morphologically different phases under a single word, like vanam in section 4. additionally, in the options for selecting or rejecting the words, sometimes the depth of the graph goes so deep that, there is a chance to miss some solutions. here we would like to mention that some of the phase information is still relevant for segmentation. and this corresponds to the compounds. the phase information tells if something is a component of a compound or a standalone noun. there are a few phases such as iic, iif, etc. that we do not ignore. barring these we ignore all other phases. therefore, we propose the following modifications in the segmenter: ## observations the test data contained on the whole 21,127 short sandhied expressions, which were taken from various texts available at the shmt corpus. this data was a parallel corpus of sandhied and unsandhied expressions. in case there are more than one segmentation possible, only one segmentation that was appropriate in the context where the sandhied expression was found is recorded. the above data was fed to both the old and the modified segmenters. the results of the old segmenter were used as the baseline. a comparison was done on how the updated system performed with respect to the old system. the correct solution's position in the old segmenter was compared with the correct solution's position in the updated segmenter. the old segmenter was able to correctly produce the segmented form in 19,494 cases out of the 21,127 instances. of these, 53.51% of the solution was found to be in the first position, 12% in second position, and 9.61% in the third. all put together, 75.12% of the correct solutions were found in the top three solutions. another important observation was that, the entire number of solutions taken all together was 2,40,942 for 21,127 test instances and the average number of solutions was 11.4 with the correct solution's position averaging at 4.71. the modified segmenter was able to correctly produce the segmented form in 19,494 cases, same as the old segmenter. and 89.27% of the solution was found to be in the first position, 6.83% in the second position, and 2.2% in the third. all put together, 98.3% of the correct solutions were found in the top three solutions. this has an increase of 23.18% from the existing system. also, the entire number of solutions taken all together was 1,46,610 for 21,127 test instances, having a drastic reduction of 94,332 solutions. the average number of solutions was 6.94, with the correct solution's position averaging at 1.18. it can be noted that the overall recall was 0.92270554267 for both the machines. since only the statistics have been altered, the new system doesn't provide new solutions. rather, it has increased the chances of getting the solution at the top three by 23.18%. as we observe, the updated system reduces the total amount of solutions and brings up the most likely solutions. also, we have more than 90% recall in both the cases. the missed out instances were either due to morphological unavailability or owing to the failure of the engine. once the morphological analyzer is updated, there will definitely be a boost in the efficiency. ## conclusion there are a few observations to be noted. first, by just using the pop (product of products) of the word and transition probabilities, we are able to obtain 98% precision. with better probabilities, we will definitely have better results. second, this system can now be used to mechanically split the continuous texts like saṃhita-pāṭha of the vedas or any other classical text to obtain the corresponding pada-pāṭha, which may be manually checked for