B angla TL it: A Benchmark Dataset for Back-Transliteration of R omanized B angla (2024)

Abstract: Low-resource languages like Bangla are severely limited by the lack of datasets. Romanized Bangla texts are ubiquitous on the internet, offering a rich source of data for Bangla NLP tasks and extending the available data sources. However, due to the informal nature of romanized text, they often lack the structure and consistency needed to provide insights. We address these challenges by proposing: (1) BanglaTLit, the large-scale Bangla transliteration dataset consisting of 42.7k samples, (2) BanglaTLit-PT, a pre-training corpus on romanized Bangla with 245.7k samples, (3) encoders further-pretrained on BanglaTLit-PT achieving state-of-the-art performance in several romanized Bangla classification tasks, and (4) multiple back-transliteration baseline methods, including a novel encoder-decoder architecture using further pre-trained encoders. Our results show the potential of automated Bangla back-transliteration in utilizing the untapped sources of romanized Bangla to enrich this language. The code and datasets are publicly available: https://github.com/farhanishmam/BanglaTLit.

## introduction in recent years, we have witnessed remarkable progress in various natural language processing (nlp) tasks driven by large language models (llms). however, these advancements have not been equally shared across all languages @xcite , particularly low-resource languages like bangla, despite its 250 million native speakers globally. a prevalent form of bangla text is romanized bangla, which uses phonetically similar latin scripts to represent bangla syllables. the widespread use of romanized bangla on social media and online platforms, largely due to the familiarity with english keyboard layouts such as qwerty, presents a valuable data source for lowresource languages @xcite . despite its ubiquity, significant challenges remain in processing romanized bangla, primarily due to the lack of standardized datasets. unlike other languages with complex phonetic mapping, bangla has a phonemic orthography, meaning it is written as it sounds. this characteristic simplifies romanization and adds flexibility in how bangla words can be romanized, as illustrated in figure 1 . however, the real complexity lies in the back-transliteration process, i.e., converting romanized texts back to the native bangla script, as this process must adhere to the grammatical rules of bangla. automatic back-transliteration can extend the training data of low-resource languages like bangla, as romanized texts are informal in nature and do not provide significant insights @xcite . another potential use case for automated back-transliteration is its deployment as a transliteration layer on top of any language model, enabling better interaction with romanized texts and extending the functionality of the native scripts to their romanized counterparts. current bangla transliteration datasets suffer from insufficient data samples, limited data sources, and are mostly subsets of larger multilingual datasets, as evident from ## datasets following the limitations of existing bangla transliteration datasets highlighted in ## methodology our methodology comprises two main components: i) developing a pretrained encoder for transliterated bangla and ii) employing the encoder aggregated sequence modeling. ## error analysis table-7 presents the top five most accurate and inaccurate predictions produced by the tb-xlm_r+banglat5_nmt model on our test set. for the incorrect predictions, the model learns the literal word representation of the romanized sentences, which conflicts with the annotated representation of the transliteration. we also analyzed the category-wise model performance, based on bleu score, of the xlm_r+banglat5_nmt model on our test set. figure 5 shows the distribution of the bleu score for each category. the model demonstrates strong performance in the hacking, request, help, and disapproval categories while struggling with the appreciation, apology, and religious categories. we hypothesize that the model performs poorly in the above categories due to inconsistent spelling, varied use of diacritics, phonetic representations, idiomatic expressions, slang, and contextdependent language. for example, the word for "thank you" might appear as "tnx", "10x", "tenq", "10q", "dhonnobad", and religious greetings like "আসসালামু আলাইকু ম" (peace be upon you) can have multiple transliterations, such as "assalamu alaikum" and "as-salamu alaykum". this flexibility in romanization makes it challenging for the model to learn consistent patterns and accurately translate these texts, unlike other straightforward categories like hacking and help. when comparing the outputs of gpt-4 turbo and llama-3-8b, we found gpt-4 turbo processing better back-transliteration capabilities than llama-3-8b. as seen in appendix ## conclusion we propose a large-scale bangla transliteration dataset and a romanized bangla pre-training corpus. experiments conducted on several baselines, including a novel dual encoder-decoder model architecture, show promising results in the task of romanized bangla back-transliteration. expanding the dataset to include more samples can be beneficial in training larger models or fine-tuning llms. besides, transliteration of bangla regional dialects and methods based on parameter-efficient fine-tuning of llms can be explored in the future. our research opens new doors of expansion for low-resource languages like bangla.