N ep BERT a: N epali Language Model Trained in a Large Corpus (2022)

Abstract: Nepali is a low-resource language with more than 40 million speakers worldwide. It is written in Devnagari script and has rich semantics and complex grammatical structure. To this date, multilingual models such as Multilingual BERT, XLM and XLM-RoBERTa haven’t been able to achieve promising results in Nepali NLP tasks, and there does not exist any such a large-scale monolingual corpus. This study presents NepBERTa, a BERT-based Natural Language Understanding (NLU) model trained on the most extensive monolingual Nepali corpus ever. We collected a dataset of 0.8B words from 36 different popular news sites in Nepal and introduced the model. This data set is 3 folds times larger than the previous publicly available corpus. We evaluated the performance of NepBERTa in multiple Nepali-specific NLP tasks, including Named-Entity Recognition, Content Classification, POS Tagging, and Sequence Pair Similarity. We also introduce two different datasets for two new downstream tasks and benchmark four diverse NLU tasks altogether. We bring all these four tasks under the first-ever Nepali Language Understanding Evaluation (Nep-gLUE) benchmark. We will make Nep-gLUE along with the pre-trained model and data sets publicly available for research.

## introduction in recent years, especially in the last four years, there has been a lot of progress in the field of nlp, which includes two breakthroughs: the selfattention mechanism @xcite and the self-supervised model pre-training @xcite , which uses the advantage of pre-training on huge volume of unlabeled text dataset. to obtain a state of the art result, a large model based on the transformer @xcite is pre-trained on a large amount of unlabeled text data, then this model is further fine-tuned * equal contributions; part of the work was done when sulav and milan were at ioe, pashchimanchal campus, nepal with labeled data as per the requirement. since its release in 2019, bidirectional encoder representations from transformers (bert) @xcite has become very popular for transfer learning purposes in various nlp tasks. many improvements of bert @xcite @xcite have been made since 2019, even though only two versions of bert which were pre-trained in english and chinese language were released initially. after a while, a new version named multilingual bert @xcite was released. this model, trained in 104 languages, showed impressive performance on many languages specific downstream tasks. some of its performances are still state-of-the-art in many languages. multilingual bert's strong performance inspired many nlp communities to build their language-specific bert model. some of the popular monolingual bert models are russian @xcite , @xcite , arabic @xcite , french @xcite and portuguese @xcite . nepali is spoken by more than 40 millions people worldwide. syntactically, nepali language differs compared to english which is one of the most widely studied languages. generally, in english the sentence structure is subject -verb -object. whereas, in nepali language this structure ends with verb having standard structure as subject -object -verb as shown in figure1. we suggest the readers refer @xcite for more information. since nepali is considered a low-resource language @xcite , it has received little attention in the field of nlp. despite the advancement of nlp in the english language, there has not been a considerable contribution to the nepali nlp domain. the main reason behind this is a lack of pre-training data, resource standardization, and computational resources. nepali is written in the devnagari script, which has been rarely used for nlp services. motivated by the success of language-specific models over multilingual models in many other languages, we present nepberta, a bert @xcite based nepali language model. the data required to pre-train nepberta were collected through the scrapping of the top 36 news sites of nepal in the nepali language. inspired by the use case of the glue @xcite ) benchmark, we also introduce the nepali natural language understanding (nlu) dataset on two downstream tasks (news content classification and categorical pair similarity) and evaluate nepberta on altogether four diverse downstream tasks on, pos tagging, news content classification, named entity recognition, and categorical pair similarity. we have brought all these tasks under nepali language understanding evaluation benchmark (nep-glue) tasks. ## related work in 2013 a team at google led by thomas mikolov released a word embedding named "word2vec" @xcite . following the success of word2vec, other forms of word embeddings like glove @xcite and fasttext @xcite were released. however, these embeddings were not able to extract the contextual meaning of the sentence. this problem was overcome by the large pre-trained models such as ulmfit @xcite , bert @xcite , roberta @xcite , xl-net @xcite , and albert @xcite . ulmfit uses a recurrent neural network as its core, whereas bert uses a self-attention mechanism, which evaluates the dependency of a to-ken with every other token in the same sequence. bert adopts the mask language modeling (mlm) technique and next sentence prediction (nsp) technique to learn the deeper semantics and contextual information of a sentence. later, @xcite and @xcite investigated the potential of bert on crosslingual nlp tasks using a large corpus of diverse languages. their work established the benchmark for many multilingual tasks and demonstrated that a single model can learn from numerous languages. in terms of model size and performance, xlm @xcite and xlm-roberta @xcite ) made significant advances. there have already been various monolingual models that outperformed multilingual ones. some of these models are finbert @xcite for finish, bertje @xcite and robbert @xcite for dutch, flaubert @xcite for french. recently two monolingual nepali models trained in the nepali language corpus were made open source on github foot_0 foot_1 . these two models were mainly trained on text corpus made available by the oscar @xcite dataset, which is more than 3 times smaller than our dataset. furthermore, there were not any benchmarks to evaluate the performance of those models across various downstream tasks. ## nepali language understanding evaluation (nep-glue) benchmark several individuals have studied nepali nlp tasks and contributed to them. parts of speech tagging @xcite , named entity recognition @xcite , and so on are examples. however, there has not been a unified, comprehensive study of the nepali nlu tasks. other languages, such as english @xcite , french @xcite , and korean @xcite , have language-specific benchmark systems for certain activities. text categorization, sequence labeling, and text span prediction are the three types of nlu tasks in general. as a result, we have developed four distinct tasks for the nep-glue benchmark. all of the codes and dataset foot_2 for these activities are freely available to the public for future usage and improvement. ## conclusion and future works until now, students and researchers were compelled to use multilingual models for their work. we introduced nepberta, a nepali language model that can be used for many fine-tuning tasks in the future. we also introduce the first-ever nepali language understanding evaluation bench-mark. in the future, we will be adding more downstream tasks in nep-glue. after the introduction of the language model in the nlp community, this will be the first time the nepali nlp community will be benefited to a great extent. we believe that the introduction of nep-berta in nepali nlp community will promote more study and implementation of the language model for many downstream tasks. there is always room for improvement in any research activity. likewise, our next plan as an improvement to this version is to increase the pre-training model size and data. sites, each of which contributes more than 100,000 data points to our corpus, include ekantipur.com, onlinekhabar.com, nagariknews.com, thahakabar.com, setopati.com,reportersnepal.com, etc. each news portal has a particular domain of interest, like hamrokhelkud.com, which publishes sports news ranging from the ipl, nba, formula 1, mma, etc., which helps us create a corpus having a diverse range of domains.