Development of A ssamese Rule based Stemmer using W ord N et (2019)

Abstract: Stemming is a technique that reduces any inflected word to its root form. Assamese is a morphologically rich, scheduled Indian language. There are various forms of suffixes applied to a word in various contexts. Such inflected words if normalized will help improve the performance of various Natural Language Processing applications. This paper basically tries to develop a Look-up and rule-based suffix stripping approach for the Assamese language using WordNet. The authors prepare the dictionary with the root words extracted from Assamese WordNet and Named Entities. Appropriate stemming rules for the inflected nouns, verbs have been set to the rule engine and later tested the stemmed output with the morphological root words of Assamese WordNet and Named Entities by computing hamming distance. This developed stemmer for the Assamese language achieves accuracy of 85%. Also, the authors reported the IR system’s performance on applying the Assamese stemmer and proved its efficiency by retrieving sense oriented results based on the fired query. Thus, Morphological Analyzer will embark the research wing for developing various Assamese NLP applications.

## introduction computationally, stemming is the process to automatically extract the base form of a given inflected word. the stemmed word is not required to be identical with the morphological root of the word. most indian languages are highly inflectional and many words in a document appear in many morphological forms. indexing is the important subtask of an ir system. indexing all words in a document appearing in various morphological forms is highly tedious and time-consuming. thus, it is necessary to stem the words to reduce them to their original base form. reducing to their original base form will help the indexer in ir to detect the important terms in a document, detect named entities, multi-word expression and extract stopwords. looking deeply into the matter, we found that two parts-of-speech nouns and verbs have a wide list of inflections for the assamese language. the main objective of this paper is to perform stemming task on a group of inflected words to retrieve root words with an acceptable accuracy. many approaches to stemming have been identified. they are classified into three categories-rule-based, statistical and hybrid approaches. rule-based approach-such approaches apply a set of morphotactic rules of a language to an inflected word. such rules may derive the base form by emitting the suffix or the prefix. statistical approach-one of the drawbacks of rule-based approach is that it is language dependent and it is dependent on the database. statistical approach overcomes both the problems by calculating probabilistic distributions of the terms. hybrid approach-combination of both rulebased and statistical approaches. in this paper, the authors have researched and implemented a rule-based stemmer for assamese language embedding the look-up based approach. the quick look-up approach is made on the dictionary prepared from assamese wordnet and named entities. assamese wordnet is a large lexical knowledge database developed by the team @xcite . it contains four major components- ## background work this section gives us an overview of stemmers developed in indian languages. for the english language, the most commonly used stemming algorithm is the porter stemming algorithm @xcite ) which followed a rule-based approach. the indian language @xcite @xcite @xcite @xcite in which stemmer is developed along with the approaches used and accuracies derived is mentioned in ## stemmer in assamese ir information retrieval system retrieves relevant and sense oriented information to a user based on the query. assamese nlp aims to develop a monolingual search engine which will help the web users to retrieve information in ones own native language say assamese. only a few (2-3) percent of people of assamese community knows to speak, read or write english, so retrieving information in own language will be much benefited. assamese ir system is technically composed of two parts-apache solr & nutch. apache solr is an open source search platform written in java from apache lucene project. some of the major features of solr are-full text search, real time indexing, dynamic clustering etc. apache nutch is also a java coded tool with the crawler feature. the crawler can be biased to fetch important relevant pages at first. we developed assamese monolingual system considering solr3.4 and nutch1.4 as indexer and crawler respectively. stemming is an important plug-in of ir. stemming is performed to an inflected word to avoid mismatches between words that share the same root word. let us consider a simple example-if we are searching for a document entitled ways to write a book and the user issues a query writing, than there will be no match with the title. but, if the query is stemmed before than the search system will stem the word writing to write and the retrieval will become easier and successful. stemming is applied to both query processing module and ir system module. both at the indexing time and during processing of the query the stemmer module is added as plug-in to assamese ir system. here, we have analyzed the performance of ir system based on two categories-• ir performance without stemming ## conclusions the performance of the assamese stemmer mentioned in this paper shows that it attains a state of art accuracy as a stand along system as well as a component of information retrieval system. the proposed technique is dictionary look-up and rule-based approach for this indo-aryan language with an acceptable accuracy of 85% and 22 defined morphotactic rules.increasing the dictionary size will result in more increasing accuracy. assamese stemmer is the basic language resource and is used in many applications in the field of text mining and nlp like ir, mt, document classification etc. the accuracy of the stemmer can be improved by defining more stemming rules and increasing the dictionary size with more root words. moreover, as the ir performance on performing stemming to the inflected terms indicates an overwhelming result, thus stemmer is an important resource for assamese nlp.