CNN for Modeling S anskrit Originated B engali and H indi Language (2022)

Abstract: Though recent works have focused on modeling high resource languages, the area is still unexplored for low resource languages like Bengali and Hindi. We propose an end to end trainable memory efficient CNN architecture named CoCNN to handle specific characteristics such as high inflection, morphological richness, flexible word order and phonetical spelling errors of Bengali and Hindi. In particular, we introduce two learnable convolutional sub-models at word and at sentence level that are end to end trainable. We show that state-of-the-art (SOTA) Transformer models including pretrained BERT do not necessarily yield the best performance for Bengali and Hindi. CoCNN outperforms pretrained BERT with 16X less parameters and achieves much better performance than SOTA LSTMs on multiple real-world datasets. This is the first study on the effectiveness of different architectures from Convolution, Recurrent, and Transformer neural net paradigm for modeling Bengali and Hindi.

## introduction bengali and hindi are the fourth and sixth most spoken language in the world, respectively. both of these languages originated from sanskrit @xcite and share some unique characteristics that include (i) high inflection, i.e., each root word may have many variations due to addition of different suffixes and prefixes, (ii) morphological richness, i.e., there are large number of compound letters, modified vowels and modified consonants, and (iii) flexible word-order, i.e., the importance of word order and their positions in a sentence are loosely bounded (examples shown in figure 1 ). many other languages such as nepali, gujarati, marathi, kannada, punjabi and telugu also share these characteristics. neural language models (lm) have shown great promise recently in solving several key nlp tasks such as word prediction and sentence completion in major languages such as english and chinese @xcite @xcite @xcite . to the best of our knowledge, none of the existing study investigates the efficacy of recent lms in the context of bengali and hindi. we conduct an in-depth analysis of major deep learning architectures for lm and propose an end-to-end trainable memory efficient cnn architecture to address the unique characteristics of bengali and hindi. state-of-the-art (sota) techniques for lm can be categorized into three sub-domains of deep learning: (i) convolutional neural network (cnn) @xcite (ii) recurrent neural network @xcite @xcite , and (iii) transformer attention network @xcite @xcite . long short term memory (lstm) @xcite based models, which are suitable for learning sequence and word order information, are not effective for modeling bengali and hindi due to their flexible word order characteristic. on the other hand, transformers use dense layer based multi-head attention mechanism. they lack the ability to learn local patterns in sentence level, which in turn puts negative effect on modeling languages with loosely bound word order. most importantly, neither lstms nor transformers use any suitable measure to learn intra-word level local pattern necessary for modeling highly inflected and morphologically rich languages. we observe that learning inter (flexible word order) and intra (high inflection and morphological richness) word local patterns is of paramount importance for bengali and hindi lm. to accommodate such characteristics, we design a novel cnn architecture, namely coordinated cnn (cocnn) that achieves sota performance with low training time. in particular, cocnn consists of two learnable convolutional sub-models: word level (vocabulary learner (vl)) and sentence level (terminal coordinator (tc)). vl is designed for syllable pattern learning, whereas tc serves the purpose of word coordination learning while maintaining positional independence, which suits the flexible word order of bengali and hindi. cocnn does not explicitly incorporate any self attention mechanism like transformers; rather it relies on tc for emphasizing on important word patterns. cocnn achieves significantly better performance than pretrained bert for bengali and hindi lm with 16x less parameters. we further enhance cocnn by introducing skip connection and parallel convolution branches in vl and tc, respectively. this modified architecture (with negligible increase in parameter number) is named as cocnn+. we validate the effectiveness of cocnn+ on a number of tasks that include next word prediction in erroneous setting, text classification, sentiment analysis and spell checking. cocnn+ shows superior performance than contemporary lstm based models and pretrained bert. in summary, the contributions of this paper are as follows: ## related works although a significant number of works for lm of high resource languages like english and chinese are available, very few researches of significance for lm in low resource languages like bengali and hindi exist. in this section, we mainly summarize major lm related research works. sequence order information based statistical rnn models such as lstm and gru have been popular for lm tasks @xcite . @xcite showed the effectiveness of lstm for english and french lm. the regularizing effect on lstm was investigated by @xcite . sota lstm models learn sub-word information in each time stamp. @xcite proposed a morphological information oriented character n-gram based word vector representation. it was improved by @xcite and is known as fasttext. @xcite proposed a technique for learning sub-word level information from data, while such an idea was integrated in a character aware lstm model by @xcite . @xcite further improved word representation by combining ordinary word level and character-aware embedding. @xcite showed that characteraware neural lms outperform syllable-aware ones. @xcite evaluated such models on 50 morphologically rich languages. self attention based transformers have become the sota mechanism for sequence to sequence modeling in recent years @xcite . some recent works have explored the use of such models in lm. deep transformer encoders outperform stacked lstm models @xcite . a deep stacked transformer model utilizing auxiliary loss was proposed by @xcite for character level language modeling. the multi-head self attention mechanism was replaced by a multi-linear attention mechanism with a view to improving lm performance and reducing parameter number @xcite . bengali and hindi language, having unique characteristics, remain open as to what strategy to use for model development in such domains. one dimensional version of cnns have been used recently for text classification oriented tasks @xcite @xcite . @xcite studied cnn application in lm showing the ability of cnns to extract lm features at a high level of abstraction. furthermore, dilated conv was employed in bengali lm with a view to solving long range dependency problem @xcite . ## conclusion we have proposed coordinated cnn (cocnn) that introduces two 1d cnn based key concepts: word level vl and sentence level tc. detailed investigation in three deep learning paradigms (cnn, lstm and transformer) shows the effectiveness of cocnn in bengali and hindi lm. we have also shown a simple but effective enhancement of cocnn by introducing skip connection and parallel conv branches in the vl and tc portion, respectively. future research may incorporate interesting ideas from existing sota 2d cnns in cocnn.