  Pre-training language models on large volume of data with self-supervised
objectives has become a standard practice in natural language processing.
However, most such state-of-the-art models are available in only English and
other resource-rich languages. Even in multilingual models, which are trained
on hundreds of languages, low-resource ones still remain underrepresented.
Bangla, the seventh most widely spoken language in the world, is still low in
terms of resources. Few downstream task datasets for language understanding in
Bangla are publicly available, and there is a clear shortage of good quality
data for pre-training. In this work, we build a Bangla natural language
understanding model pre-trained on 18.6 GB data we crawled from top Bangla
sites on the internet. We introduce a new downstream task dataset and benchmark
on four tasks on sentence classification, document classification, natural
language understanding, and sequence tagging. Our model outperforms
multilingual baselines and previous state-of-the-art results by 1-6%. In the
process, we identify a major shortcoming of multilingual models that hurt
performance for low-resource languages that don't share writing scripts with
any high resource one, which we name the `Embedding Barrier'. We perform
extensive experiments to study this barrier. We release all our datasets and
pre-trained models to aid future NLP research on Bangla and other low-resource
languages. Our code and data are available at
https://github.com/csebuetnlp/banglabert.
