Punctuation and case restoration in code mixed I ndian languages (2022)

Abstract: Automatic Speech Recognition (ASR) systems are taking over in different industries starting from producing video subtitles to interactive digital assistants. ASR output can be used in automatic indexing, categorizing, searching along with normal human readability. Raw transcripts from ASR systems are difficult to interpret since it usually produces text without punctuation and case information (all lower, all upper, camel case etc.), thus limiting the performance of downstream NLP tasks. We proposed an approach to restore the punctuation and case for both English and Hinglish (i.e Hindi vocabulary in Latin script) languages. We have performed a classification task using encoder-based transformers which is a mini BERT consisting of 4 encoder layers for punctuation and case restoration instead of the traditional Seq2Seq model considering the latency constraint in real world use cases. It consists of a total number of 15 distinct classes for the model which includes 5 punctuations i.e Period(.), Comma(,), Single Quote(‘), Double Quote(”) & Question Mark(?) with different combinations of casing. The model is benchmarked on an internal dataset which was based on user conversation with the voice assistant and it achieves a F1(macro) score of 91.52% on the test set.

## introduction raw transcripts from asr systems are difficult to interpret and not very user friendly for display purposes.to make the asr transcripts more readable and interpretable, we need to include appropriate punctuation and segmentation at word and sentence level. we have experimented and pivoted to a unique word level classification approach with certain techniques of model optimizations making it useful in real time. punctuation are marks used in printed and written documents to separate sentences and clauses and to help make the meaning of sentences more clear.the standard english punctuation is as follows: period, comma, apostrophe, quotation, question, exclamation, brackets, braces, parenthesis, dash, hyphen, ellipsis, colon, semicolon. auto punctuation and capitalization is a way to automatically add punctuation and restore casing to a sentence thereby making it suitable to read for users. example 1: raw text : lets eat shyam converted text : let's eat, shyam. example 2: raw text : shyam khaane chalein converted text : shyam, khaane chalein? example 3: raw text : hello astor how are you converted text : hello astor, how are you? implementing auto-punctuation and capitalization on asr output can improve its readability, have better display and help improve several downstream nlp tasks such as, * neural machine translation * sentimental analysis * text summarization * named entity recognition ## motivation in recent years, studies on asr have shown outstanding results but there are still difficulties in standardizing the output of asr @xcite such as capitalization and punctuation restoration for speech transcriptions. the problems restrict readers to understand the asr output semantically and also cause difficulties for natural language processing models such as ner, pos and semantic parsing. in this paper, we propose a method to restore the punctuation and case for asr transcription. most of the punctuation and case restoration models work in seq2seq (encoder-decoder) neural network architectures like t5, bart, gpt etc. although these models are very good at generating long text sequences based on the task they are trained and fine tuned for, this comes with a challenge of high latency, which is a bottleneck for real time asr systems. to address this challenge, we have framed the task as a classification problem. ## experiment details we have approached the punctuation and case restoration task as a text classification problem where there are a total of 18 possible combinations of punctuation and casing, out of which we have considered 15 unique classes. currently, our model supports 5 types of punctuation i.e period (represented as p), comma (c), question mark (?), single quote (sq) and double quote (dq) & 3 types of casing i.e lower cased (represented as oth-ers), upper cased (all_cap) and sentence cased (cap_init). we have given a few examples of each category as mentioned in ## model architecture our base model is a pre-trained bert-mini model which has 4 bert encoder layers. we have wired two linear layers on top of it as a classification head for the word level text classification. this could process a maximum 256 tokens in one sequence of text[2]. the main purpose of using bert-encoder is, it is faster in comparison to any seq2seq model and the context of words is learnt better which helps us understand the patterns of the language. a glimpse of how bert[3] works is shown in figure 1 . the training of the task is done in two phases : -1. pre-training : the original sentence is usually passed to bert and then tokenized us-ing the word piece encoder, which generates contextual -embeddings i.e the embeddings depend on the context . transformer reads the entire sequence of words based on its surroundings from both directions simultaneously instead on left to right/ right to left @xcite . e.g : for the sentence, "i have a pen do you", the corresponding punctuation labels for it is predicted as, "cap_init, others, oth-ers, p, cap_init, q" respectively. ## training details for model training purposes, textual data from publicly available ncert textbooks along with prepared in-house data was used. approximately 500000 sentences were used as training data which were cleaned and formatted to get rid of noisy data and make it suitable for a machine learning model. it consists of 15 unique labels i.e 'all_cap', 'all_cap_c', 'all_cap_p', 'all_cap_q', 'all_cap_sq', 'c', 'cap_init', 'cap_init_c', 'cap_init_p', 'cap_init_q', 'cap_init_sq', 'others', 'p', 'q' & 'sq'. since we are using a supervised learning technique, input data (lower case with removed punctuation) and their corresponding labeled data were fed to the model. we have performed the complete experiment in one tesla v100 gpu system, which got 16 gb of memory. some of the hyper-parameters used in the training are as follows: to make the model output be as close as possible to the desired output or ground truth values. during model training, the model weights are adjusted iteratively to minimize the loss. cross entropy loss is popularly used in classification tasks both in machine learning and deep learning @xcite . cross-entropy is defined in figure 3 . ## model optimization model optimization helps us in achieving below objectives. ## results we prepared an internal testing dataset with 2050 data which was based on user conversation with the voice assistant. on a sentence level evaluation, the performance of our model on the test set is shown in in the table, the correctly predicted sentences is referred as true and the incorrectly predicted sentences is referred as false which has an accuracy score of 82%. ## observations since our problem statement is framed as a classification task we have only used the encoders. we were able to reduce the computational power to half and reduce the latency significantly. considering our model is trained on both english and romanized hindi, there are some words which are spelled the same but mean completely different in different sentences which could cause ambiguity. here's an example below. sentence 1 : do you know me? sentence 2 : do apple chahiye. although both sentences start with "do", sentence 1 should end with a question mark ('?') while sentence 2 should end with a period('.'). we have trained the model with sentences using maximum possible ambiguous words in different contexts to handle these challenges due to the code mix. after benchmarking our test dataset, we observed that out of all the labels used, 'c' seems to be difficult to predict and place in the right position which could be due to less training data with commas. we could revisit the data preparation phase and include more sentences with "," in different positions and evaluate the model. ## limitation there are a few limitations to our model. first being, not able to evaluate our model on any public dataset due to lack of resources in hinglish data for auto-punctuation domain. due to lack of hardware resources, our current model is limited to 32 tokens which is approximately 25 words in hindi. ## future work we would improve our existing model through the following steps. ## conclusion we present an approach to restore punctuation and case of the raw output obtained from the asr system with significantly reduced latency. currently our model is trained on english and hinglish (i.e hindi vocabulary in english script) data and achieves expected performance under different conditions.