A Rule Based Lightweight B engali Stemmer (2020)

Abstract: In the field of Natural Language Processing (NLP) the process of stemming plays a significant role. Stemmer transforms an inflected word to its root form. Stemmer significantly increases the efficiency of Information Retrieval (IR) systems. It is a very basic yet fundamental text pre-processing task widely used in many NLP tasks. Several important works on stemming have been carried out by researchers in English and other major languages. In this paper, we study and review existing works on stemming in Bengali and other Indian languages. Finally, we propose a rule based approach that explores Bengali morphology and leverages WordNet to achieve better accuracy. Our algorithm produced stemming accuracy of 98.86% for Nouns and 99.75% for Verbs.

## introduction information retrieval is a very essential process to extract relevant data or documents systematically from big data collections. inverted index is a crucial data structure used in almost all modern ir systems. all the words in the entire data collection is stemmed first before the inverted index is built. thus stemming plays a very important role in ir @xcite . stemming is the process of reducing inflectional or derived variant forms of one word to its root form. two major components of ir task is basically indexing and retrieval. stemming not only enhances the recall rate of ir task but also reduces the index size significantly. thus, it increases the efficiency of the information retrieval system. depending on the linguistic rules of a particular language, words of any natural language can be inflected in many ways. bengali is one of the most morphologically decorated languages. generally inflected words are generated from the root word by adding some suffixes. it is observed that one root word in bengali may have more than 20 morphological variants. another challenge is in finding root word from a compound word i.e a word can be formed by conjunction of more than one root words. a large number of notable works have been done on stemming techniques for different languages from the last couple of decades. most of the approaches were first applied in english language and subsequently adapted in other languages. different stemming approaches involve different techniques such as longest suffix matching, dictionary based look-up, co-occurrence computation etc. several works @xcite @xcite have been done based on these techniques in different languages. however, a very few papers utilize these techniques in bengali language but failed to achieve over 95% of accuracy. in this paper, we propose a rule based technique that utilizes rich volume of bengali grammar rules also involves bengali word-net @xcite to achieve better accuracy. it is worth mentioning that we also verify the extracted stem word from our rule based algorithm with the help of modified wordnet. the sole purpose of the modification of the wordnet here is to reduce the complexity of the approach by circumventing the unnecessary prefix suffix removal of the word. the verification process and wordnet modification thus help us to achieve next level of higher accuracy. the following example helps to understand the proper usage of the wordnet in this approach. suppose a word সরাইখানা (inn) is the root word and within this word one of the suffixes খানা is present. if we remove the suffix from the word, it will mean something else. similar example can be seen for the word ব ার (strong) where one of the suffix set র is present within the word. in bengali stemming, filtering of root word and inflected word was a immense challenge. we overcome this problem by introducing the wordnet in the methodology. in the subsequent section, we present several cases where the modification of wordnet plays an important role. the rest of the paper is structured as follows: section 2 states the related works in this literature and brief review on them. formation of different types of inflections in bengali languages are analyzed in section 3. section 4 elaborates different techniques involved in the proposed methodology and the entire methodology. different resources that are used in the evaluation of the algorithm is described in section 5. experiment and dataset used in the experiment is detailed out in section 6. evaluation and results are presented in section 7. finally we conclude this paper in section 8. ## related work there are two types of approaches in stemming mechanism. the first approach is called stemming in which affixes and suffixes are removed in order to extract the root word. the second approach is called lemmatization. lemmatization requires a sound knowledge of the researcher in the particular language. as an example the word "good" in english language has its variants "better" and "best". in order to get the lemma (root word) "good" we need lemmatization. lemmatization involves dictionary look up to solve this kind of unusual case of extracting root words. stemming technique for english language is well studied and several techniques have been developed. the very first stemming technique lovins stemmer was proposed by @xcite . lovins stemmer removes a suffix just in two major steps. it maintains 294 endings, 29 conditions and 35 transformation rules. later, well known rule based porters stemming @xcite technique was proposed in 1980 which is basically a rule based algorithm. in its 8 steps it handles different cases of morphological and inflexional endings in different steps. an extension of the lovins algorithm is known as dawson stemmer @xcite was proposed which covers wide range of list of 1200 suffixes. these above mentioned algorithms work very well for english language but we are more interested in different stemming algorithms for indian languages. @xcite proposed a lightweight stemmer for hindi in 2004 which removes the suffixes based on the longest suffix matching from a list of suffixes. they also developed a suffix list in hindi language to enhance the performance of the stemming. @xcite proposed an affix-exception list based stemmer for urdu language. they omit prefix and suffix from the word based on looking up the exception list of prefixes or suffixes. this stemmer finds the stem word based on lexical look up method. a successful look up ignores the stripping off of the prefix and suffix of a word. hussain et al. proposed a stemming mechanism for urdu language based on n-gram stripping model @xcite . @xcite developed a brute force algorithm to strip off suffixes in order to find stem words in punjabi language. they have overcome the problem of over-stemming and under-stemming. the suffix stripping is replaced sometimes by suffix substitution. @xcite proposed a lightweight stemmer which strips off suffixes and finds the stem word for bengali. the fundamental idea of this algorithm is to remove suffixes based on the longest suffix matching. they also maintain a list of possible suffixes for bengali language. @xcite reported a simple corpus based unsupervised stemmer for bengali language. the algorithm uses some statistics collected from corpus analysis based on the co-occurrences between word variants. they generate a graph where nodes are the variants of word and an edge between them represents a co-occurrence. @xcite used the porter stemming technique in bengali language. @xcite developed a stemming technique based on statistical clustering based approach to discover equivalence classes of root words using some set of distance @xcite shows the highest accuracy of 96.27% for bengali language with respect to other indian languages and the lowest accuracy is 80.73% for punjabi language proposed by @xcite . ## inflections of words in bengali language inflection of word is a process in which a word takes different forms. it may be based on tense, number and person. different forms of the actual word are called inflected words. bengali is one of the most morphologically decorated languages due to its wide range of inflected words. in this language inflections are mainly observed for verbs and nouns. adjectives in bengali can take only two suffixes -তর and তম, marking comparative and superlative adjectives, respectively, while adverbs in bengali do not get suffixed. in this section we address the inflections of verbs and nouns. we also analyze the rules of inflections. ## proposed methodology we have discussed about various verb and noun inflections in bengali language till now. we also mentioned possible suffixes for deviation of a word and the different root-verb of different lengths of verbs. in this section, we illustrate our methodology to find stem of a verb. it is easily observable that the number of rules for formation of different length of rootটা, টােক, িট, টার, র, টােত, িটেত, ◌র, ◌ etc েলা, িল, েলার, িলর, এরা, জন, িলেত, েলােত, রািজ, রািশ, পু , সমূ হ, বণ , বৃ দ, বগ , মালা, িদ etc verbs are limited. we can generate root-verbs of different length from an inflected word by applying the mentioned rules. for example from the inflected word খেয়েছন (ate) we can generate possible root-verbs of different length upto 3. if we consider length 3 we get খয়ছা, খু য়ছা, িখয়ছা etc about 6 words. similarly for length 2 we can generate words like খয ় , খহ,খা , িখয়, খু etc about 8 words and finally for length 1 we have খ, খা, িখ, খু . out of these all possible root-verbs only one will be matched with valid one (খা) and corresponding verb(খাওয়া) (eat) will be retrieved. at this point we would like to illustrate our proposed methodology for stemming. the algorithm is presented in algorithm 1. this algorithm takes one word possibly an inflected word i w and access bengali wordnet. before doing any kind of suffix removal it checks the word in the wordnet to confirm that the word is inflected or not. in some cases suffixes are present in a word and creates a new word. removing this suffix from the word changes the intended meaning of the word. for example আধার (dark), if we remove 'র' from the word, it will be আধা (half or half pieces). on the other hand if we consider word গাধার (donkey's) and if we remove `র' then it will be গাধা (donkey) which is valid suffix removal. so before doing suffix removal we should search that particular word in the wordnet. in the next stage, algorithm perform the stemming mechanism according to its category (noun or verb). in this phase root form of the verb is generated. the function g() generates the root-verb according to the lengths ranging from 3 to 1. it generates all possible root-verbs based on the first 3 letters of the inflected word by applying rules mentioned in outputafter applying the following procedure actual root word will be assigned to r w . we define an array of possible suffixes for noun words mentioned in another major notable limitation of bengali wordnet is that, wordnet does not contain deviated words a lot. it is certainly much difficult to identify all linguistic deviation of a word and store them all in wordnet. a word can be spoken or written in different way and that is deviated from one region to another region of a country. as an example, the deviated form of the word বলা (telling) can be বলেত or বলতা. inhabitants of various regions of west bengal (a state of india) use the word 'বলেত' and on the other hand some inhabitants of some regions of west bengal use the word 'বলতা'. at this point we briefly illustrate the steps involved in the proposed algorithm. the algorithm takes inflected word (i w ) as input. in the next step function p() extracts the parts of speech (w pos ) of i w . if it is a "noun" then a function s() searches the noun database (n d ) for a match. if a match found, it means the inflected word has a meaning itself hence there is no need of stemming. otherwise we can strip off different suffixes depending upon the presence of suffixes within the inflected word. we iterate through the suffix set s f and a function r() searches different possible suffixes within the inflected word and removes them. r() returns a word rw p in every iteration that does not contain i th suffix set of s f . rw p then searched in the "root noun" database r nd using function s(). whenever a match found the word is returned as the root form of the inflected word. if w pos is "verb" then again the search function s() searches the word in verb database (v d ) for a match. a match indicates that the word has its own meaning. so we return the word as it was. otherwise a function g() will generate all possible root-verb from the inflected word and store them in set rv ð . in first iteration it takes first 3 characters of the inflected word and make all possible root-verbs using the predefined rules. then it takes 2 characters and so on. as in bengali grammar root-verbs can have of length maximum of three, we start finding root-verbs in descending order. every element within the set rv ð is checked in the root-verb database r vd . at any point if a match found then the corresponding stem verb (r w ) is returned. ## resources used bengali wordnet: bengali wordnet is a part of indowordnet 1 . bengali wordnet is a lexical database for bengali words and it contains around 61 thousand bengali words along with the synsets. we imported the bengali wordnet in mysql database. we created a separate table for all the root-verbs corresponding to the bengali verbs. tdil corpus: for the corpus, we used the technology development for indian languages (tdil) corpus @xcite in this work. ## conclusions and future work we have proposed a rule based algorithm for stemming verbs and nouns in bengali. incorporation of wordnet adds an extra degree in validation and extracting root words from in-flected words. bengali grammar rules have been used to find root-verbs of verbs efficiently. we have covered almost all kinds of root-verbs and possible suffixes to create a root word from an inflected verb.