Advancing B angla Punctuation Restoration by a Monolingual Transformer-Based Method and a Large-Scale Corpus (2023)

Abstract: Punctuation restoration is the endeavor of reinstating and rectifying missing or improper punctuation marks within a text, thereby eradicating ambiguity in written discourse. The Bangla punctuation restoration task has received little attention and exploration, despitethe rising popularity of textual communication in the language. The primary hindrances in the advancement of the task revolve aroundthe utilization of transformer-based methods and an openly accessible extensive corpus, challenges that we discovered remainedunresolved in earlier efforts. In this study, we propose a baseline by introducing a mono-lingual transformer-based method named Jatikarok, where the effectiveness of transfer learning has been meticulously scrutinized, and a large-scale corpus containing 1.48M source-target pairs to resolve the previous issues. The Jatikarok attains accuracy rates of 95.2%, 85.13%, and 91.36% on the BanglaPRCorpus, Prothom-Alo Balanced, and BanglaOPUS corpora, thereby establishing itself as the state-of-the-art method through its superior performance compared to BanglaT5 and T5-Small. Jatikarok and BanglaPRCorpus are publicly available at: https://github.com/mehedihasanbijoy/Jatikarok-and-BanglaPRCorpus

## introduction the continuous effort to bridge the linguistic gap between human natural language and digital devices has propelled natural language processing (nlp) to its current level of advancement. despite these advances in nlp, bangla language processing continues to present significant challenges including multimodal complexities stemming from intricate language rules. proper punctuation placement, particularly in the bangla language, plays a pivotal role in further reducing this barrier and 1 যিতকারক facilitating downstream bangla natural language processing (bnlp) tasks. previous studies have highlighted the dominance of transformer-based models such as bert @xcite , roberta @xcite , and albert @xcite and have showcased their efficacy in leveraging contextual information for punctuation restoration in high-resource languages. additionally, architectural enhancements such as attention mechanisms has also shown good performance @xcite . following the trend of domain-specific fine-tuning of pretrained models, alongside post-processing techniques, has also achieved close to adequate performance @xcite . cross-lingual augmentation strategies enhance transformer models for languages with diverse resources, which is nonexistent in @xcite . the study conducted by @xcite only restored four types of punctuation marks in the bangla language. however, there are at least nine more punctuation marks that need to be addressed to exhaustively capture the meaning. moreover, it should be pointed out that deep learning models may not be capable of covering a significant proportion of punctuation in cases where the corpus is comparably small @xcite . however, transformer-centric approaches have started demonstratring impressive performance in various bnlp tasks, including grammar and spelling error correction @xcite . surprisingly, transformer-based methods have yet to be applied in any studies for the bangla punctuation restoration task. consequently, in this study, we leverage the impressive capabilities of transformers and initiate an investigation into their unexplored potential for the task. in this study, we propose a transformer-based method named jatikarok for the task with a uniquely tailored architecture of six encoder and decoder layers, optimizing the balance between model complexity and computational efficiency for bangla punctuation restoration, while enhancing its performance through transfer learning, which consequently renders it a monolingual method. furthermore, we introduce banglaprcorpus, a large-scale parallel corpus for the task consisting of 1.48 million source-target pairs. the contributions of this paper are summarized below: the subsequent sections of the paper are organized as follows: section 2 presents an in-depth analysis of the background of bangla punctuation restoration; the process of constructing our corpus is expounded upon in section 3; section 4 elucidates the architecture of our proposed method; section 5 presents the tangible results derived from our empirical study; section 6 culminates our investigation by offering concluding remarks and outlining potential avenues for future research. ## literature review the task of punctuation restoration has garnered widespread attention, leading to the emergence of novel insights within methods and datasets. we delve into an examination of the recent studies conducted for punctuation restoration. our extensive studies identified several contemporary transformer-based and deep learning methods in the realm of bangla punctuation restoration tasks and in various high and low-resource languages such as transformer (lai et al., @xcite , @xcite and hybrid @xcite . among rnn-based methods @xcite proposed a novel approach comprised of a bidirectional recurrent neural network (brnn) model with an attention mechanism. the authors trained a large bangla dataset focusing specifically on predicting the exclamation mark and achieved 96.8% accuracy with various post-processing techniques. likewise, @xcite took a similar approach to solve the task. the advent of nlp has seen the employment of transformer-based methods where m-bert, bert, roberta, biobert, and electra have been utilized @xcite . @xcite explored transformerbased language models to restore punctuation and improved bangla training and evaluation data whereas @xcite utilized inadequate supervision and proposed a unique method for acquiring dialogue data in languages with few resources and evaluated the dataset by finetuning banglabert @xcite . predicting punctuation for sequences instead of individual tokens by utilizing roberta-base, @xcite proposed an innovative approach to solving the task. @xcite followed a homogeneous approach and proposed a contextual embedding-based punctuation prediction model. roberta outperformed other transformer-based models in the comparison. our study has also revealed that the implementation of hybrid models has yielded exceptional results in addressing complex natural language processing tasks. by taking advantage of the evaluation of different bert transformer models using lstm and gru with a linear neural network layer @xcite proposed a robust punctuation restoration algorithm. besides, @xcite proposed a lstm-crf(conditional random field) model that uses pre-trained bert embeddings to make tagging decisions that take step interdependence into account to solve the punctuation restoration problem. a thorough analysis found that transformerbased methods outperform rnn-based ones. while rnns may struggle with feature coverage and handling large datasets, transformers do not face these challenges. however, a downside of transformer-based approaches is that they require huge datasets to perform effectively. ## corpus creation we consider a total of 16 distinct punctuation marks, including period('।'), comma(','), exclamation mark ('!'), question mark ('?'), semicolon (';'), bangla colon ('◌ঃ'), colon (':'), double quotation mark ("), single quotation mark ('), hyphen ('-'), opening parenthesis ('('), closing parenthesis (')'), opening curly brace ('{'), closing curly brace ('}'), opening square bracket ('['), and closing square bracket(']'), to curate the corpus. the details of these punctuation marks are delineated as follows: period(।): a definitive halt, denoting the terminus of a sentence in bangla. comma(,): an eloquent separator, orchestrating rhythm within lists, crafting succinct pauses, and clarifying sentence structure. ## conclusion this study addressed the primary obstacle hindering the progress of the task by introducing a comprehensive baseline. specifically, we introduced the groundbreaking jatikarok, a monolingual transformer-based method meticulously designed to harness the power of transfer learning by adapting knowledge from bangla grammatical error correction to effectively tackle intricate linguistic patterns inherent to this specific task. furthermore, the efficacy of our proposed jatikarok is validated across various corpora, solidifying its status as a state-of-the-art method for this task by outperforming banglat5 and t5-small. in conjunction with the model, a substantial parallel corpus containing 1.48m source-target pairs has been made publicly accessible, which has been carefully curated by incorporating 16 bangla punctuation marks. consequently, this resource eliminates the scarcity of materials for bangla, effectively transforming it into a language well-equipped for punctuation tasks. in our future study, we will empirically investigate the effectiveness of knowledge distillation through the transfer of knowledge from the multilingual model to our monolingual model.