P un K tuator: A Multilingual Punctuation Restoration System for Spoken and Written Text (2021)

Abstract: Text transcripts without punctuation or sentence boundaries are hard to comprehend for both humans and machines. Punctuation marks play a vital role by providing meaning to the sentence and incorrect use or placement of punctuation marks can often alter it. This can impact downstream tasks such as language translation and understanding, pronoun resolution, text summarization, etc. for humans and machines. An automated punctuation restoration (APR) system with minimal human intervention can improve comprehension of text and help users write better. In this paper we describe a multitask modeling approach as a system to restore punctuation in multiple high resource – Germanic (English and German), Romanic (French)– and low resource languages – Indo-Aryan (Hindi) Dravidian (Tamil) – that does not require extensive knowledge of grammar or syntax of a given language for both spoken and written form of text. For German language and the given Indic based languages this is the first towards restoring punctuation and can serve as a baseline for future work.

## introduction automatic speech recognition (asr) has become ubiquitous these days and has wide applications in business and personal life. one of the drawbacks of asr is it produces an unpunctuated stream of text. restoring punctuation manually is a timeconsuming task. apart from spoken text a large amount of written text online -blogs, articles, social media,etc. -sometimes lack the appropriate punctuation marks due to human inconsistencies, which can alter the meaning of text. an apr system designed with an understanding of asr and written forms of text can help resolve these issues. transcriptions passed to an apr system, can improve the following machine learning tasks such as machine translation, conversational agents, coreference resolution, etc. further it can be used as an unsupervised auxiliary or pretext task, for training large scale transformer language models, as it would require understanding about global structure of the text. prior punctuation restoration methods have mostly been solved using lexical features, prosodic features or combination of both. due to large availability of text data, majority of the methods have focused on using lexical features. early methods @xcite used hidden markov models (hmm) to model punctuation using acoustic features such as pause duration, pitch and intensity. though the acoustic based models perform well on asr system, they can perform better when combined with textual data. @xcite ; @xcite ; @xcite proposed various methods that combined lexical features along with prosodic information thereby improving apr tasks. @xcite proposed unidirectional and bidirectional long short term memory (bi-lstm) based punctuation prediction model which did not require extensive feature engineering. though the above method considered the long distant token dependencies, it ignored label dependencies. to address label dependencies @xcite made use of recurrent neural networks for sequence to sequence mapping using an encoder-decoder architecture. recently the use of transformer based approaches combination of speech and pre-trained word embeddings have achieved state of art performance on iwslt datasets (spoken transcripts from ted talks for asr tasks, but often used as benchmark for comparison of punctuation restoration models). yi et al. (2020) used pretrained bert @xcite that is used to perform adversarial multi-task learning to restore punctuation. and used an augmentation strategy to make models more robust to asr errors. though most approaches have shown considerable improvement in overcoming some of the challenges faced in terms of modeling and achieving the state of performance in spoken language transcripts in english, there are the following limitations: to overcome some of the challenges, we make the following contributions: ## conclusion in this paper we described and implemented a joint modeling approach for restoring punctuation for high and low resource languages across spoken and written text. joint language model trained with auxiliary language and text mode classification improved the performance of the apr task. we achieved reasonable performance on the benchmark iwslt datasets without being trained on it. we also presented a web extension that can help multilingual users improve overall readability and coherence of text. further we present baseline results on indic languages that can be used for future work. we have shown examples of punctuated text that was output from our system in the appendix section. ottokar tilk and tanel alumäe. 2016. bidirectional recurrent neural network with attention mechanism for punctuation restoration. in interspeech, pages 3047-3051. eva vanmassenhove and christian hardmeier. 2018. europarl datasets with demographic speaker information. jie yang and yue zhang. 2018. ncrf++: an opensource neural sequence labeling toolkit. arxiv preprint arxiv:1806.05626. jiangyan yi and j. tao. 2019. self-attention based model for punctuation prediction using word and speech embeddings. icassp 2019 -2019 ieee international conference on acoustics, speech and signal processing (icassp), pages 7270-7274. jiangyan yi, jianhua tao, ye bai, zhengkun tian, and cunhang fan. 2020. adversarial transfer learning for punctuation restoration. arxiv preprint arxiv:2004.00248.