Punctuation Restoration using Transformer Models for High-and Low-Resource Languages (2020)

Abstract: Punctuation restoration is a common post-processing problem for Automatic Speech Recognition (ASR) systems. It is important to improve the readability of the transcribed text for the human reader and facilitate NLP tasks. Current state-of-art address this problem using different deep learning models. Recently, transformer models have proven their success in downstream NLP tasks, and these models have been explored very little for the punctuation restoration problem. In this work, we explore different transformer based models and propose an augmentation strategy for this task, focusing on high-resource (English) and low-resource (Bangla) languages. For English, we obtain comparable state-of-the-art results, while for Bangla, it is the first reported work, which can serve as a strong baseline for future work. We have made our developed Bangla dataset publicly available for the research community.

## introduction due to the recent advances in deep learning methods, the accuracy of automatic speech recognition (asr) systems has increased significantly (e.g., 3.4% wer on librispeech noisy test set @xcite ). the improved performance of asr enabled the development of voice assistants (e.g., siri, cortana, bixby, alexa, and google assistant) and their wider use at the user end. among different components (e.g., acoustic, language model), pre-and post-processing steps, the punctuation restoration is one of the post-processing steps that also needs to be dealt with to improve the readability and utilize the transcriptions in the subsequent nlp applications @xcite . 1 this is because state-of-the-art nlp models are mostly trained using punctuated texts (e.g., texts from newspaper articles, wikipedia). hence, the lack of punctuation significantly degrades performance. for example, there is a performance difference of more than ∼ 10% when the model is trained with newspaper texts and tested with transcriptions for the named entity recognition system @xcite . to address this issue, most of the earlier efforts on the punctuation restoration task have been done using lexical, acoustic, prosodic, or a combination of these features @xcite @xcite @xcite . for the punctuation restoration task, lexical features have been widely used because the model can be trained using any punctuated text (i.e., publicly available newspaper articles or content from wikipedia) and because of the availability of such large-scale text. this is a reasonable choice as developing punctuated transcribed text is a costly procedure. in terms of machine learning models, conditional random field (crf) has been widely used in earlier studies @xcite . lately, the use of deep learning models, such as long short-term memory (lstm), convolutional neural network (cnn), and transformers have also been used @xcite @xcite for this task. there has been a variant of transformer based language models (e.g., bert @xcite , roberta @xcite ), which have not been explored widely to address this problem. hence, we aimed to explore different architectures and finetune pre-trained models for this task focusing on english and bangla. punctuation restoration models are usually trained on clean texts but used on noisy asr texts. as such, the performance may degrade due to errors introduced by asr models which are not present in the training data. we design an augmentation strategy (see section 4.1.2) to address this issue. for english, we train and evaluate the models using iwslt reference and asr test datasets. we report that our proposed augmentation strategy yields a 3.8% relative improvement in the f1 score on asr transcriptions for english and obtains state-of-the-art results. for bangla, there has not been any prior reported work for punctuation restoration. in addition, no resource has been found. therefore, we prepare a training dataset from a news corpus and provide strong baselines for news, reference, and asr transcriptions. to shade light in the current state-of-the-art on punctuation restoration task, our contributions in this study are as follows: ## related work recent lexical features based approaches for punctuation restoration tasks are predominantly based on deep neural networks. @xcite used pre-trained word embeddings to train feedforward deep neural network and cnn. their result showed improvements over a crf based approach that uses purely text data. since context is important for this type of task, several studies explored the recurrent neural network (rnn) based architectures combined with crf and pre-trained word vectors. for instance, tilk and alumäe (2016) used a bidirectional recurrent neural network (rnn) with an attention mechanism to improve performance over dnn and cnn models. in another study, @xcite used character-level lstm architecture to achieve results that are competitive with the wordlevel crf based approach. @xcite combined bidirectional lstm with a crf layer and an ensemble of three networks. they further used knowledge distillation to transfer knowledge from the ensemble of networks to a single dnn network. transformer based approaches have been explored in several studies @xcite . @xcite combined pretrained word and speech embeddings that improves performance compared to only word embedding based model. @xcite used transformer architecture to restore both punctuation and capitalization. punctuation restoration is also important for machine translation. the study by @xcite used a transformer based model for spoken language translation. they achieved significant improvements over cnn and rnn baselines, especially on joint punctuation prediction task. more recent approaches are based on pre-trained transformer based models. @xcite used pre-trained bert @xcite model with bidirectional lstm and a crf layer to achieve state-of-the-art result on reference transcriptions. @xcite used adversarial multitask learning with auxiliary parts of speech tagging task using a pre-trained bert model. in this study, we also explore transformer based models; however, unlike prior works that solely studied one architecture (bert), we experiment with different models. we also propose a novel augmentation scheme that improves the performance. our augmentation is closely related to the augmentation techniques proposed in @xcite where the authors consider synonym replacement, random insertion, random swap, and random deletion. while their work is intended for the text classification tasks, we propose a different version of it for this study, which is a sequence labeling task. we do not use synonym replacement and random swap as they do not usually appear in speech transcription. ## experiments for this study, we explored different transformer based models for both english and bangla. in addition, we used bidirectional lstm (bilstm) on top of the pre-trained transformer network, and an augmentation method to improve the performance of the models. ## conclusion in this study, we explore different transformer models for high-and low-resource languages (i.e., english and bangla). in addition, we propose an augmentation technique, which improves performance on noisy asr texts. there has not been any reported result and resources for punctuation restoration on bangla. our study, findings, and developed resources will enrich and push the current state-of-art for this low-resource language. we have released the created bangla dataset and code for the research community.