Improving Punctuation Restoration for Speech Transcripts via External Data (2021)

Abstract: Automatic Speech Recognition (ASR) systems generally do not produce punctuated transcripts. To make transcripts more readable and follow the expected input format for downstream language models, it is necessary to add punctuation marks. In this paper, we tackle the punctuation restoration problem specifically for the noisy text (e.g., phone conversation scenarios). To leverage the available written text datasets, we introduce a data sampling technique based on an n-gram language model to sample more training data that are similar to our in-domain data. Moreover, we propose a two-stage fine-tuning approach that utilizes the sampled external data as well as our in-domain dataset for models based on BERT. Extensive experiments show that the proposed approach outperforms the baseline with an improvement of 1.12% F1 score.

## introduction asr systems are widely deployed in many products to support a variety of business applications, such as customer support, virtual assistants, and conversational systems. the context for this research is to build commercial products that perform realtime transcription of english business telephone calls, with additional natural language processing to enable features such as assistance for customer support agents and note-taking during business meetings. though asr systems trained on large amounts of data can transcribe human speech with high accuracy @xcite , spoken dialogue transcripts produced by such systems often exclude punctuation marks. therefore, the user reading a long unpunctuated transcript may find it unreadable @xcite . beyond the need for readability, downstream natural language understanding (nlu) models use transcripts to provide more business insights. many nlu models (e.g., sentiment analysis and named entity recognition (ner)) achieving state-of-theart performance are based on large-scale language models (e.g. bert @xcite ), which are often pre-trained on well-punctuated texts such as english wikipedia and bookcorpus @xcite . however, nlu models trained on such datasets may not perform well on unpunctuated transcripts. thus, for the conversational domain, punctuated speech transcripts are also necessary. while most prior research has focused on automatically punctuating monologue speech transcripts, our work is focused on transcripts of realworld phone conversations with turn-taking between two or more people. spontaneous turntaking conversations typically are more noisy than monologues because they might contain speech overlaps, false starts, repetitions, and higher transcription word error rates. one way to address this problem is to obtain large amounts of humanpunctuated transcripts as training data. however, human annotation is expensive and timeconsuming. in this paper, we propose a novel way to leverage freely available external data to improve the punctuation restoration accuracy in human-tohuman business conversational data. the main contributions of this paper are as follows: ## related work in this section, we first review the recent work on punctuation restoration. since we sample additional training data from external corpora, we also review related work on data sampling. ## method we frame punctuation restoration as a sequence labeling problem. there are four classes that the model needs to predict, including pe-riod foot_0 , comma, questionmark, and none. more specifically, the model predicts what the punctuation mark next to a given token is. if there is no punctuation mark after the input token, the prediction should be none. in figure 1 , we show the overall architecture of our proposed model that utilizes a two stage finetuning approach: first, (i) we do data sampling from an external dataset (e.g., movie subtitles) using our proposed n-gram similarity approach and fine-tune a pre-trained bert model. then, (ii) we annotate unpunctuated phone conversation transcripts generated from an asr system to construct an in-domain training set (noisy data) to apply the second stage of fine-tuning using the bert model fine-tuned in the previous stage. below, we describe the data sampling, data annotation, and twostage fine-tuning approach in detail. ## experiments in our experiments, we investigate the following: (i) will the model perform better if we use external datasets? (ii) will the model performance degrade if we reduce the number of layers? below, we first describe our experimental settings. then we analyze the model performance when external data are used, followed by discussing the performance when layer reduction is applied. ## conclusion in this paper, we propose a new approach to sample training data that are similar to the target domain from freely available general corpora via an n-gram language model. we conduct extensive experiments using bert with different data settings for the punctuation restoration task and find that our two-stage fine-tuning approach is effective to improve model performance. we also show the effectiveness of our model for real-time inference as we find that removing the first 6 layers of the 12 layer bert does not harm the accuracy but substantially improves the speed. in the future, we will investigate how other external datasets @xcite impact the model performance.