B an L emma: A Word Formation Dependent Rule and Dictionary Based B angla Lemmatizer (2023)

Abstract: Lemmatization holds significance in both natural language processing (NLP) and linguistics, as it effectively decreases data density and aids in comprehending contextual meaning. However, due to the highly inflected nature and morphological richness, lemmatization in Bangla text poses a complex challenge. In this study, we propose linguistic rules for lemmatization and utilize a dictionary along with the rules to design a lemmatizer specifically for Bangla. Our system aims to lemmatize words based on their parts of speech class within a given sentence. Unlike previous rule-based approaches, we analyzed the suffix marker occurrence according to the morpho-syntactic values and then utilized sequences of suffix markers instead of entire suffixes. To develop our rules, we analyze a large corpus of Bangla text from various domains, sources, and time periods to observe the word formation of inflected words. The lemmatizer achieves an accuracy of 96.36% when tested against a manually annotated test dataset by trained linguists and demonstrates competitive performance on three previously published Bangla lemmatization datasets. We are making the code and datasets publicly available at https://github.com/eblict-gigatech/BanLemma in order to contribute to the further advancement of Bangla NLP.

## introduction lemmatization is a crucial task in natural language processing (nlp), where the goal is to obtain the base form of a word, known as the lemma. it has widespread applications in several nlp tasks, such as information retrieval @xcite ), text classification @xcite , machine translation @xcite , etc. lemmatization is a particularly challenging task in highly inflectional languages such as bangla @xcite , due to the large number of inflectional and derivational suffixes that can be added to words. generally, lemmatization reduces the inflectional form of a word to its dictionary form. lemmatization in bangla has several challenges due to various linguistic factors. firstly, the language exhibits a wide range of morphological diversity, making it difficult for a system to cover all its aspects @xcite . secondly, bangla has approximately 50000 roots, each capable of generating a large number of inflected words based on several factors such as tense, gender, and number @xcite . thirdly, bangla words can have multiple meanings, known as polysemy, depending on their part-of-speech (pos), surrounding words, context, and other factors @xcite . additionally, the development of lemmatization systems for this complex morphological language is hindered by the lack of available resources. prior research attempts have employed different methodologies (e.g., learning-based, rule-based, and hybrid approaches) @xcite . despite the fact that some studies have shown satisfactory performance in specific scenarios, there remains a pressing need for a robust lemmatizer tailored to the bangla language. in this study, we have taken a rule and dictionary-based approach to tackle the lemmatization problem in bangla. unlike other rule-based studies for lemmatization in bangla, we have derived the stripping methods based on the suffix marker sequences considering the pos class of a word. a suffix েদরগুেলােত (d̪ ergulot ̪e) from a word িশশুেদরগুেলােত (ʃiʃud̪ ergulot ̪e) is a combination of েদর (d̪ er), গুেলা (gulo), and েত (t ̪e) markers. we strip the last to the first marker sequentially to ob- figure 1 : figure 1a shows the illustration of our proposed banlemma lemmatizer. for simplicity, we demonstrate noun lemmatization only. for noun, emphasis, case, and plural markers are stripped in succession as specified in section 3.1. figure 1a shows its effectiveness on cross-dataset settings. on @xcite , we outperform their lemmatizer by a significant margin and achieve competitive results on the whole dataset of @xcite . on @xcite , we outperform their model on the corrected pos and lemma version of their evaluation set, elaborated in section 5.3. contextual lemmatizers lemmatize a word based on the surrounding context using deep neural networks. @xcite employed a two-stage bidirectional gated recurrent neural network to predict lemmas without using additional features. release of the universal dependencies (ud) dataset @xcite and sigmorphon 2019 shared task formed the basis of encoder-decoder architectures to solve the lemmatization task as a string-transduction task @xcite . for the bangla language, @xcite employed a similar two-step attention network that took morphological tags and inflected words as input. @xcite used pos tags of each word as additional features to the encoder-decoder network achieving 95.75% accuracy on validation dataset. earlier rule-based approaches did not consider the composition of suffixes and removed them based on the highest length or trie-like data structures. in contrast, we provide specific rules on how a sequence of markers forms a suffix based on a word's pos tag and show its efficacy on a crossdataset setup. ## methodology in bangla, the meaning of a word is greatly influenced by its pos class within a given context of a sentence (see ## experimental setup we evaluate banlemma's performance using different pos taggers: human-annotated tags, bnlp toolkit @xcite , and isi 4 using the stanford postagger 5 implementation. additionally, we conduct cross-dataset evaluation and compare our methodology with existing bangla lemmatizers, including benlem @xcite , banel @xcite , and @xcite . ## conclusion and future works this study introduces banlemma, a bangla lemmatization system aimed at enriching bangla language resources. banlemma is composed of linguistically derived rules, obtained through rigorous analysis of a large bangla text corpus. to overcome the challenges associated with limited resources in bangla lemmatization, we also provide a comprehensive collection of morphological markers and rules. to demonstrate the effectiveness of banlemma, we have conducted evaluations using a human-annotated test dataset, annotated by trained linguists and some recently published bangla datasets. our proposed banlemma achieved an accuracy of 96.36% on our humanannotated test set. furthermore, in cross-dataset evaluation, banlemma exhibited significant performance improvements ranging from 1% to 11%. the results of our study shed light on the formation of inflected words, offering a solution to address the limitations of previous lemmatization methods. these findings contribute to the advancement of research in this field and pave the way for further investigations in the domain of bangla lemmatization.