Contextual B angla Neural Stemmer: Finding Contextualized Root Word Representations for B angla Words (2023)

Abstract: Stemmers are commonly used in NLP to reduce words to their root form. However, this process may discard important information and yield incorrect root forms, affecting the accuracy of NLP tasks. To address these limitations, we propose a Contextual Bangla Neural Stemmer for Bangla language to enhance word representations. Our method involves splitting words into characters within the Neural Stemming Block, obtaining vector representations for both stem words and unknown vocabulary words. A loss function aligns these representations with Word2Vec representations, followed by contextual word representations from a Universal Transformer encoder. Mean Pooling generates sentence-level representations that are aligned with BanglaBERT’s representations using a MLP layer. The proposed model also tries to build good representations for out-of-vocabulary (OOV) words. Experiments with our model on five Bangla datasets shows around 5% average improvement over the vanilla approach. Notably, our method avoids BERT retraining, focusing on root word detection and addressing OOV and sub-word issues. By incorporating our approach into a large corpus-based Language Model, we expect further improvements in aspects like explainability.

## introduction large language models (llms) like bert @xcite , gpt @xcite , and others have proven their efficacy in various natural language processing (nlp) tasks. they excel at capturing contextual information and cultural subtleties in specific languages. these models exhibit strong capabilities for addressing diverse nlp tasks, especially during their unsupervised pretraining phase. however, in low resource language like bangla, there are so many language specific problems that haven't been resolved yet since bangla language lack comprehensive lexicons, word embeddings, or linguistic resources. firstly, there may be a good number of out-ofvocabulary (oov) words which may hamper the nlp tasks. secondly, in llms, tokenizing one word can result splitting into different subwords that make the model difficult to explain. in the following paragraphs, we clarify these problems with examples. in second set of problems are observed due to the use of bangla stemmer/lemmitizer. there are many existing way for finding the root words like stemming and lemmitizer. lemmitizer needs ground truth word mapping to find the word. on the other hand, stemming algorithms typically use heuristics to identify the suffixes of words that can be removed to obtain the root form. however, by re- method tokens original text নটরেডম, হিলকৰ্স ও িভকারুনিনসা কেলেজ ভিতর্ হেত পারেব না ধূ মপায়ী িশক্ষাথীরা। banglabert tokenizer ['নট', '##র', '##েডম', ',', 'হিল', '##কৰ্', '##স', 'ও', 'িভক', '##◌ার', '##◌ু ন', '##িন', '##সা','কেলেজ' 'ভিতর্ ', 'হেত', 'পারেব', 'না', '[unk]', 'িশক্ষাথীরা' , '।'] it has been shown that splitting words into multiple subwords is not the best option all the time. there are some cases where this might not work well @xcite , @xcite . some words might have a prefix or suffix that changes the meaning of the word, but bert's subword tokenizer might split it into separate subwords. for example, in considering the aforementioned limitations of rule based stemmer, we want to create a contextual bangla neural stemmer for bangla language to find better representation of words. especifically, in our proposed method, by splitting each word of a sentence into characters in neural stemming block, we will get vector representation for not only the stem word but also the unknown vocab word. a loss is used to make sure that the representations of the words should be aligned with the word2vec representations. then after a liner layer transformations those representations is passed into universal transformer (ut) @xcite encoder to assure of getting contextual representation of a word via self attention. mean pooling is used to get a sentence level representation for a sentence from those contextual word representations. a mlp layer is used and a loss is defined to align the sentence representation with the banglabert's one. the whole model pipeline is described in section 3. our model employs character-based representations to find root word representations, which effectively addresses the issues of oov tokens and subword tokenization. by combining these representations with bert, our model is capable of obtaining contextual representations for these root words, enhancing its ability to capture the semantic nuances and context of the language. we evaluate our model performance in 5 different bangla dataset. in every dataset, our model outperforms the vanilla approach by a good margin (around 5% improvement on average). more details are described in section 5. please note that our goal is not retraining the bert at all. instead of retraining the bert, our proposed method is used finding the root words with contextual representations and address oov & sub-word problem. if we create a llm based on our methodology with a large corpus, our method may outperform the vanilla bert in different aspects and may improve the explainability also. therefore, the summary of the contributions of this paper is given below. ## methodology the tokenizers that are used in transformer based model like bpe, wordpiece, unigram split a word into multiple subwords. this may cause information discrepancy between the actual meaning of the actual word which may affect the low resource language models like bangla language model. besides, there also may have a good amount of out of vocabulary words in those low resource language model. one option is to find the root form of the words but this approach miss the contexutal information. considering all scenarios, we propose a character based contextual neural stemmer which not only find the stemmed root word to surpass subword techniques but also give the contextual embeddings. for being character based model, our proposed model can also deal with the out of vocabulary issues. our model have two major components, character level neural stemming block, universal transformer encoding block along with two different losses for fulfilling our criteria. ## conclusion in this research, we proposed a contextual bangla neural stemmer to overcome the limitations of traditional rule-based stemmers. by obtaining vector representations for both stem words and unknown vocabulary words, our method offers improved word representations for bangla language processing tasks. the model leverages the uni-versal transformer encoder and mean pooling to capture contextual word and sentence-level representations. our evaluation on five bangla datasets demonstrated significant performance gains, outperforming the vanilla approach. notably, our approach focuses on root word detection and addressing oov and sub-word problems rather than retraining the bert. our findings suggest that a large corpus-based language model incorporating our methodology could further enhance nlp tasks and potentially improve explainability. by addressing the limitations of stemmers and providing better word representations, our proposed approach opens new avenues for research in bangla language processing and contributes to advancing natural language understanding in the context of bangla text.