B angla BERT : Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in B angla (2022)

Abstract: In this work, we introduce BanglaBERT, a BERT-based Natural Language Understanding (NLU) model pretrained in Bangla, a widely spoken yet low-resource language in the NLP literature. To pretrain BanglaBERT, we collect 27.5 GB of Bangla pretraining data (dubbed ‘Bangla2B+’) by crawling 110 popular Bangla sites. We introduce two downstream task datasets on natural language inference and question answering and benchmark on four diverse NLU tasks covering text classification, sequence labeling, and span prediction. In the process, we bring them under the first-ever Bangla Language Understanding Benchmark (BLUB). BanglaBERT achieves state-of-the-art results outperforming multilingual and monolingual models. We are making the models, datasets, and a leaderboard publicly available at https://github.com/csebuetnlp/banglabert to advance Bangla NLP.

## introduction despite being the sixth most spoken language in the world with over 300 million native speakers constituting 4% of the world's total population, 1 bangla is considered a resource-scarce language. @xcite categorized bangla in the language group that lacks efforts in labeled data collection and relies on self-supervised pretraining @xcite @xcite to boost the natural language understanding (nlu) task performances. to date, the bangla language has been continuing to rely on fine-tuning multilingual pretrained language models (plms) @xcite @xcite . however, since multilingual plms cover a wide range of languages @xcite , they are large (have hundreds of millions of parameters) and require substantial computational resources for fine-tuning. they also tend to show degraded performance for low-resource languages @xcite on downstream nlu tasks. motivated by the triumph of language-specific models @xcite ; @xcite ; @xcite ; @xcite , inter alia) over multilingual models in many other languages, in this work, we present banglabert -a bert-based @xcite bangla nlu model pretrained on 27.5 gb data (which we name 'bangla2b+') we meticulously crawled 110 popular bangla websites to facilitate nlu applications in bangla. since most of the downstream task datasets for nlp applications are in the english language, to facilitate zero-shot transfer learning between english and bangla, we additionally pretrain a model in both languages; we name the model banglishbert. we also introduce two datasets on bangla natural language inference (nli) and question answering (qa), tasks previously unexplored in bangla, and evaluate both pretrained models on four diverse downstream tasks on sentiment classification, nli, named entity recognition, and qa. we bring these tasks together to establish the firstever bangla language understanding benchmark (blub). we compare widely used multilingual models to banglabert using blub and find that both models excel on all the tasks. we summarize our contributions as follows: ## the bangla language understanding benchmark (blub) many works have studied different bangla nlu tasks in isolation, e.g., sentiment classification @xcite @xcite , semantic textual similarity @xcite , parts-of-speech (pos) tagging @xcite , named entity recognition (ner) @xcite . however, bangla nlu has not yet had a comprehensive, unified study. motivated by the surge of nlu research brought about by benchmarks in other languages, e.g., english @xcite , french @xcite , korean @xcite , we establish the first-ever bangla language understanding benchmark (blub). nlu generally comprises three types of tasks: text classification, sequence labeling, and text span prediction. text classification tasks can further be sub-divided into single-sequence and sequence-pair classification. therefore, we consider a total of four tasks for blub. for each task type, we carefully select one downstream task dataset. we emphasize the quality and open availability of the datasets while making the selection. we briefly mention them below. ## experiments & results setup we fine-tuned banglabert and banglish-bert on the four downstream tasks and compared them with several multilingual models: mbert @xcite , xlm-r base and large @xcite , and indicbert @xcite , a multilingual model for indian languages; and sahajbert @xcite , an albertbased @xcite ) plm for bangla. all pre5 we removed the yes/no questions from tydiqa and subsampled the unanswerable questions to have equal proportion. ## conclusion & future works creating language-specific models is often infeasible for low-resource languages lacking ample data. hence, researchers are compelled to use multilingual models for languages that do not have strong pretrained models. to this end, we introduced banglabert and banglishbert, two nlu models in bangla, a widely spoken yet lowresource language. we presented new downstream datasets on nli and qa, and established the blub benchmark, setting new state-of-the-art results with banglabert. in future, we will include other bangla nlu benchmarks (e.g., dependency parsing @xcite ) in blub and investigate the benefits of initializing bangla nlg models from banglabert.