BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla
Abstract
In this work, we introduce BanglaBERT , a
BERT-based Natural Language Understanding (NLU) model pretrained in Bangla, a widely spoken yet low-resource language in the NLP literature.
To pretrain BanglaBERT, we collect 27.5 GB of Bangla pretraining data (dubbed ‘Bangla2B+’) by crawling 110 popular Bangla sites.
We introduce two downstream task datasets on natural language inference and question answering and benchmark on four diverse NLU tasks covering text classiﬁcation, sequence labeling, and span prediction.
In the process, we bring them under the ﬁrst-ever Bangla Language Understanding Benchmark (BLUB).
BanglaBERT achieves state-of-the-art results outperforming multilingual and monolingual models.
We are making the models, datasets, and a leaderboard publicly available at https://github. csebuetnlp/banglabert to advance Bangla NLP.
Introduction
Despite being the sixth most spoken language in the world with over 300 million native speakers constituting 4% of the world’s total population,[-
-]
Bangla is considered a resource-scarce language.
Joshi et al.
(2020b) categorized Bangla in the language group that lacks efforts in labeled data collection and relies on self-supervised pretraining (Devlin et al., 2019;
Radford et al., 2019; Liu et al., 2019) to boost the natural language understanding (NLU) task performances.
To date, the Bangla language has been continuing to rely on ﬁne-tuning multilingual pretrained language models (PLMs) (Ashraﬁ et al., 2020; Das et al., 2021; Islam et al., 2021).
However, since multilingual PLMs cover a wide range of languages (Conneau and Lample, 2019; Conneau et al., 2020), they are large (have hundreds of millions of parameters) and require substantial computational resources for ﬁne-tuning.
They also tend to show degraded performance for low-resource languages (Wu and Dredze, 2020) on downstream NLU tasks.
Motivated by the triumph of language-speciﬁc models (Martin et al.
Polignano et al.
; Canete et al.
Antoun et al.
(2020), inter alia ) over multilingual models in many other languages, in this work, we present BanglaBERT – a BERT-based
(Devlin et al., 2019)
Bangla NLU model pretrained on 27. GB data (which we name ‘ Bangla2B+ ’)
we meticulously crawled 110 popular Bangla websites to facilitate NLU applications in Bangla.
Since, most of the downstream task datasets for NLP applications are in the English language, to facilitate zero-shot transfer learning between English and Bangla, we additionally pretrain a model on both languages; we name the model BanglishBERT .
We also introduce a Bangla Natural Language
Inference (NLI) and a Question Answering (QA) dataset, tasks previously unexplored in Bangla, and evaluate BanglaBERT on four diverse downstream tasks on sentiment classiﬁcation, NLI, named entity recognition, and QA.
We bring these tasks together to establish the ﬁrst-ever B angla
L anguage U nderstanding B enchmark ( BLUB ).
We compare two widely used multilingual models to BanglaBERT using BLUB and ﬁnd that BanglaBERT excels on all the tasks.
We summarize our contributions as follows:
We present two new pretrained models: BanglaBERT and BanglishBERT; and introduce new Bangla NLI and QA datasets.
We introduce Bangla Language Understanding Benchmark (BLUB) and show that, in the supervised setting, BanglaBERT outperforms mBERT and XLM-R (base) by 6.8 and 4.3 BangLUE score, while in zero-shot crosslingual transfer, BanglishBERT outperforms them by 15.5 and 10.6, respectively.
We release code and provide a leaderboard to spur future research on Bangla NLU.
BanglaBERT 2. Pretraining Data
A high volume of good quality text data is a prerequisite for pretraining large language models.
For instance, BERT (Devlin et al., 2019) is pretrained on the English Wikipedia and the Books corpus (Zhu et al., 2015) containing 3.3 billion tokens.
Subsequent works like RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019) used more extensive web-crawled data with heavy ﬁltering and cleaning.
Bangla is a rather resource-constrained language in the web domain; for example, the Bangla
Wikipedia dump from July 2021 is only 650 MB, two orders of magnitudes smaller than the English
Wikipedia.
As a result, we had to crawl the web extensively to collect our pretraining data.
We selected 110 Bangla websites by their Amazon Alexa rankings 2 and the volume and quality of extractable texts by inspecting each website.
The contents included encyclopedias, news, blogs, e-books, stories, social media/forums, etc.[-
-]
The amount of data totaled around 35 GB.
There are also noisy sources of Bangla data dumps, a couple of prominent ones being OSCAR (Suárez et al., 2019) and CCNet (Wenzek et al., 2020).
However, they contained lots of offensive texts; we found them infeasible to clean thoroughly.
Fearing potential harmful impacts (Luccioni and
Viviano, 2021)
, we opted not to use them.
We discuss ethical considerations at the end of the paper.
2. Pre-processing
We performed thorough deduplication on the pretraining data, removed non-textual contents (e.g.,
HTML/JavaScript tags), and ﬁltered out nonBangla pages using a language classiﬁer (Joulin et al., 2017).
After the processing, the dataset was reduced to 27.5 GB in size containing 5.25M documents having 306.66 words on average.
We trained a Wordpiece (Wu et al., 2016) vocabulary of 32k subword tokens on the resulting corpus with a 400 character alphabet, kept larger than the native Bangla alphabet to capture codeswitching (Poplack, 1980) and allow romanized
Bangla contents for better generalization.
We limited the length of a training sample to 512 tokens and did not cross document boundaries (Liu et al., 2019) while creating a data point.
After tokenization, we had 7.18M samples with an average length of 304.14 tokens and containing 2.18B tokens in total; hence we named the dataset ‘ Bangla2B+ ’.
2. Pretraining Objective
Self-supervised pretraining objectives leverage unlabeled data.
For example, BERT (Devlin et al., 2019) was pretrained with masked language modeling (MLM) and next sentence prediction (NSP).
Several works built on top of this, e.g., RoBERTa (Liu et al., 2019) removed NSP and pretrained with longer sequences, SpanBERT (Joshi et al., 2020a) masked contiguous spans of tokens, while works like XLNet (Yang et al., 2019) introduced objectives like factorized language modeling.
We pretrained BanglaBERT using ELECTRA (Clark et al., 2020b), pretrained with the Replaced
Token Detection (RTD) objective, where a generator and a discriminator model are trained jointly.
The generator is fed as input a sequence with a portion of the tokens masked (15% in our case) and is asked to predict them using the rest of the input (i.e., standard MLM).
The masked tokens are then replaced by tokens sampled from the generator’s output distribution for the corresponding masks, and the discriminator then has to predict whether each token is from the original sequence or not.
After pretraining, the discriminator is used for ﬁne-tuning.
Clark et al.
(2020b) argued that
RTD back-propagates loss from all tokens of a sequence, in contrast to 15% tokens of the MLM objective, giving the model more signals to learn from.
Evidently, ELECTRA achieves comparable downstream performance to RoBERTa or XLNet with only a quarter of their training time.
This computational efﬁciency motivated us to use ELECTRA for our implementation of BanglaBERT.
2. Model Architecture & Hyperparameters
We pretrained the base ELECTRA model (a 12layer Transformer encoder with 768 embedding size, 768 hidden size, 12 attention heads,feed-forward size, generator-to-discriminator ratio 13 , 110M parameters) with 256 batch size for 2.5M steps on a v3-8 TPU instance on GCP.
We used the Adam (Kingma and Ba, 2015) optimizer with a 2e-4 learning rate and linear warmup of 10k steps.
[Table]
Metric
Domain
[Table]
SentNoB 12,575 1,567 1,567 Macro-F1 Social Media
[Table]
Language Inference BNLI 381,449 2,419 4,895 Accuracy Miscellaneous
[Table]
MultiCoNER 14,  800 Micro-F1 Miscellaneous
Question Answering
SQuAD 2.0, TyDiQA 127,771 2,502 2,504 EM/F
Wikipedia 2. BanglishBERT Often labeled data in a low-resource language for a task may not be available but be abundant in highresource languages like English. In these scenarios, zero-shot cross-lingual transfer (Artetxe and
Schwenk, 2019) provides an effective way to be still able to train a multilingual model on that task and be able to transfer to Bangla. To this end, we pretrain a bilingual model, named BanglishBERT , on Bangla and English together using the same set of hyperparameters. We use the BERT pretraining corpus as the English data and train a joint bilingual vocabulary (each language having ∼ 16k tokens). We upsample the Bangla data during training to equal participation in both languages.
The Bangla Language Understanding Benchmark (BLUB)
Many works have studied different Bangla NLU tasks in isolation, e.g., sentiment classiﬁcation (Das and Bandyopadhyay, 2010;
Sharfuddin et al., 2018; Tripto and Ali, 2018), semantic textual similarity (Shajalal and Aono, 2018), parts-of-speech (PoS) tagging (Alam et al., 2016), named entity recognition (NER)
(Ashraﬁ et al., 2020).
However, Bangla NLU has not yet had a comprehensive, uniﬁed study.
Motivated by the surge of NLU research brought about by benchmarks in other languages, e.g., English (Wang et al., 2018), French (Le et al., 2020), Korean (Park et al., 2021), we establish the ﬁrst-ever Bangla Language Understanding Benchmark (BLUB).
NLU generally comprises three types of tasks: text classiﬁcation, sequence labeling, and text span prediction.
Text classiﬁcation tasks can further be sub-divided into single-sequence and sequence-pair classiﬁcation.
Therefore, we consider a total of four tasks for BLUB.
For each task type, we carefully select one downstream task dataset.
We emphasize the quality and open availability of the datasets while making the selection.
We brieﬂy mention them below:
Single-Sequence Classiﬁcation : Sentiment classiﬁcation is perhaps the most-studied Bangla
NLU task, with some of the earlier works dating back over a decade (Das and Bandyopadhyay, 2010).
Hence, we chose this as the single-sequence classiﬁcation task.
However, most Bangla sentiment classiﬁcation datasets are not publicly available.
We could only ﬁnd two public datasets: BYSA (Tripto and Ali, 2018) and SentNoB (Islam et al., 2021).
We found BYSA to have many duplications.
Even worse, many duplicates had different labels.
SentNoB had better quality and covered a broader set of domains, making the classiﬁcation task more challenging.
Hence, we opted to use the latter.
Sequence-Pair Classiﬁcation :
In contrast to single-sequence classiﬁcation, there has been a dearth of sequence-pair classiﬁcation works in
Bangla.
We found work on semantic textual similarity (Shajalal and Aono, 2018), but the dataset is not publicly available.
As such, we curated a new
Bangla Natural Language Inference ( BNLI ) dataset for sequence-pair classiﬁcation.
We chose NLI as the representative task due to its fundamental importance in NLU.
Given two sentences, a premise and a hypothesis as input, a model is tasked to predict whether or not the hypothesis is entailment, contradiction, or neutral to the premise.
We used the same curation procedure as the XNLI (Conneau et al., 2018)
dataset: we translated the MultiNLI (Williams et al., 2018) training data using the English to Bangla translation model by Hasan et al.
(2020) and had the evaluation sets translated by expert human translators.
4 Due to the possibility of the incursion of errors during automatic translation, we used the Language-Agnostic BERT Sentence
Embeddings (LaBSE) (Feng et al., 2020) of the translations and original sentences to compute their similarity and discarded all sentences below a similarity threshold of 0.70.
Moreover, to ensure goodquality human translation, we used similar quality assurance strategies as Guzmán et al.
Sequence Labeling :
In this task, all words of a text sequence have to be classiﬁed. Named En-
Models |Params.|
SC
[Table]
NER QA BLUB Score Zero
[Table]
lingual transfer learning mBERT 180M
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
270M
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
BanglishBERT 110M
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Supervised
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
IndicBERT 18M
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
sahajBERT
[Table] [Table] BanglaBERT [Table]
[Table] [Table] [Table] [Table]
tity Recognition (NER) and Parts-of-Speech (PoS) tagging are two of the most prominent sequence labeling tasks. We chose the Bangla portion of SemEval 2022 MultiCoNER (Malmasi et al., 2022) dataset for BLUB.
Span Prediction : Extractive question answering is a standard choice for text span prediction.
Similar to BNLI, we machine-translated the
SQuAD 2.0 (Rajpurkar et al., 2018) dataset and used it as the training set.
For validation and test,
We used the Bangla portion of the TyDiQA 5 (Clark et al., 2020a) dataset.
We posed the task analogous to SQuAD 2.0: presented with a text passage and a question, a model has to predict whether or not it is answerable.
If answerable, the model has to ﬁnd the minimal text span that answers the question.
We present detailed statistics of the BLUB benchmark in Table 1.
Experiments & Results Setup
We ﬁne-tuned BanglaBERT and BanglishBERT on the four downstream tasks and compared with several multilingual models: mBERT (Devlin et al., 2019), XLM-R base and large (Conneau et al., 2020), and IndicBERT (Kakwani et al., 2020), a multilingual model for Indian languages; and sahajBERT (Diskin et al., 2021) (an ALBERTbased (Lan et al., 2020) pretrained Bangla model).
All pretrained models were ﬁne-tuned for 3epochs with batch size 32, and the learning rate was tuned from {2e-5, 3e-5, 4e-5, 5e-5}. The ﬁnal models were selected based on the validation performances after each epoch.
We performed ﬁnetuning with three random seeds and reported their average scores in Table 2. We reported the average performance of all tasks as the BLUB score.
Zero-Shot Transfer
We show the zero-shot cross-lingual transfer results of the multilingual models ﬁne-tuned on the English counterpart of each dataset (SentNob had no English equivalent; hence we used the Stanford Sentiment Treebank (Socher et al., 2013) for the zero-shot sentiment classiﬁcation task).
Here BanglishBERT achieves strong cross-lingual performance over similar-sized models but falls short of XLM-R (large) by about 1 BLUB score.
This is an expected outcome since cross-lingual effectiveness depends explicitly on model size (K et al., 2020). Supervised Fine-tuning
In the supervised ﬁnetuning setup, BanglaBERT outperformed multilingual models and monolingual sahajBERT on all the tasks, achieving a BLUB score of 77.09, even coming head-to-head with XLM-R (large).
BanglaBERT is not only superior in performance but also substantially compute- and memoryefﬁcient.
For instance, it may seem that sahajBERT is more efﬁcient than BanglaBERT due to its smaller size, but it takes 2-3.5x time and 2.43.33x memory as BanglaBERT to ﬁne-tune.
Sample efﬁciency
It is often challenging to annotate training samples in real-world scenarios, especially for low-resource languages like Bangla.
So, in addition to compute- and memory-efﬁciency, [Figure] sample-efﬁciency (Howard and Ruder, 2018) is another necessity of PLMs.
To assess the sample efﬁciency of BanglaBERT, we limit the number of training samples and see how it fares against other models.
We compare it with XLM-R (large) and plot their performances on the SC and NLI tasksfor different sample size in Figure 1.
Results show that when we have fewer number of samples ( ≤ 1 k ), BanglaBERT has substantially better performance (2-9% on SC and 6-10% on NLI with p < 0 . 05 ) over XLM-R (large), making it more practically applicable for resource-scarce downstream tasks.
Conclusion & Future Works
Creating language-speciﬁc models is often infeasible for low-resource languages lacking ample data.
Hence, researchers are compelled to use multilingual models for languages that do not have strong pretrained models.
To this end, we introduced
BanglaBERT, an NLU model in Bangla, a widely spoken yet low-resource language.
We presented a new downstream dataset on NLI and established the BLUB Benchmark, setting new state-of-the-art results with BanglaBERT.
In future, we will include other Bangla NLU benchmarks (e.g., dependency parsing (de Marneffe et al., 2021)) in BLUB and investigate the beneﬁts of initializing Bangla NLG models from BanglaBERT. Acknowledgements We would like to thank the Research and Innovation
Centre for Science and Engineering (RISE), BUET, for funding the project, and Intelligent Machines
Limited and Google TensorFlow Research Cloud (TRC) Program for providing cloud support.
Ethical Considerations Dataset and Model Release
The Copy Right Act, 2000 8 of Bangladesh allows reproduction and public release of copy-right materials for noncommercial research purposes.
As a transformative research work, we will release BanglaBERT under a non-commercial license.
Furthermore, we will release only the pretraining data for which we know the distribution will not cause any copyright infringement.
The downstream task datasets can all be made publicly available under a similar noncommercial license.
Quality Control in Human Translation Trans-
lations were done by expert translators who provide translation services for renowned Bangla newspapers.
Each translated sentence was further assessed for quality by another expert.
If found to be of low quality, it was again translated by the original translator. The sample was then discarded altogether if found to be of low quality again. Fewer thansamples were discarded in this process. Translators were paid as per standard rates in local currencies.
We tried to minimize offensive texts by explicitly crawling the sites where such contents would be nominal.
However, we cannot guarantee that there is absolutely no objectionable content present and therefore recommend using the model carefully, especially for text generation purposes.
Furthermore, we removed the personal information of the content writers by discarding the author ﬁelds while collecting the data.
Appendix Pretraining Data Sources
We used the following sites for data collection.
We categorize the sites into six types:
Encyclopedia:
• bn.banglapedia.org • bn.wikipedia.org • songramernotebook.com News: • anandabazar.com • arthoniteerkagoj.com • bangla.24livenewspaper.com • bangla.bdnews24.com • bangla.dhakatribune.com • bangla.hindustantimes.com • bangladesherkhela.com • banglanews24.com • banglatribune.com • bbc.com • bd-journal.com • bd-pratidin.com • bd24live.com • bengali.indianexpress.com • bigganprojukti.com • bonikbarta.net • chakarianews.com • channelionline.com • ctgtimes.com • ctn24.com • daily-bangladesh.com • dailyagnishikha.com • dainikazadi.net • dainikdinkal.net • dailyfulki.com • dailyinqilab.com • dailynayadiganta.com • dailysangram.com • dailysylhet.com • dainikamadershomoy.com • dainikshiksha.com • dhakardak-bd.com • dmpnews.org • dw.com • eisamay.indiatimes.com • ittefaq.com.bd • jagonews24.com • jugantor.com • kalerkantho.com • manobkantha.com.bd • mzamin.com • ntvbd.com • onnodristy.com • pavilion.com.bd • prothomalo.com • protidinersangbad.com • risingbd.com • rtvonline.com • samakal.com • sangbadpratidin.in • somoyerkonthosor.com • somoynews.tv • tbsnews.net • teknafnews.com • thedailystar.net • voabangla.com • zeenews.india.com • zoombangla.com Blogs
• amrabondhu.com • banglablog.in • bigganblog.org • biggani.org • bigyan.org.in • bishorgo.com • cadetcollegeblog.com • choturmatrik.com • horoppa.wordpress.com • muktangon.blog • roar.media/bangla • sachalayatan.com • shodalap.org • shopnobaz.net • somewhereinblog.net • subeen.com •
tunerpage.com • tutobd.com
E-books/Stories:
• banglaepub.github.io • bengali.pratilipi.com • bn.wikisource.org • ebanglalibrary.com • eboipotro.github.io • golpokobita.com • kaliokalam.com • shirisherdalpala.net • tagoreweb.in
Social Media/Forums: • banglacricket.com • bn.globalvoices.org • helpfulhub.com • nirbik.com • pchelplinebd.com •
techtunes.io
Miscellaneous:
• banglasonglyric.com • bdlaws.minlaw.gov.bd • bdup24.com • bengalisongslyrics.com • dakghar.org • gdn8.com • gunijan.org.bd • hrw.org • jakir.me • jhankarmahbub.com • jw.org • lyricsbangla.com • neonaloy.com • porjotonlipi.com • sasthabangla.com • tanzil.net
We wrote custom crawlers for each site above (except the Wikipedia dumps).
Additional Sample Efﬁciency Tests
We plot the the sample efﬁciency results of the NER and QA tasks in Figure 2.
[Figure] Similar results are also observed here for the
NER task, where BanglaBERT is more sampleefﬁcient when we have ≤ 1 k training samples. In the QA task however, both models have identical performance for all sample counts.
Compute and Memory Efﬁciency Tests
To validate that BanglaBERT is more efﬁcient in terms of memory and compute, we measured each model’s training time and memory usage during the ﬁne-tuning of each task.
All tests were done on a desktop machine with an 8-core Intel Core-i 11700k CPU and NVIDIA RTX 3090 GPU.
We used the same batch size, gradient accumulation steps, and sequence length for all models and tasks for a fair comparison.
We use relative time and memory (GPU VRAM) usage considering those of BanglaBERT as units.
The results are shown in
Table 3.
(We mention the upper and lower values of the different tasks for each model)
[Table]