Segmenting Subtitles for Correcting ASR Segmentation Errors (2021)

Abstract: Typical ASR systems segment the input audio into utterances using purely acoustic information, which may not resemble the sentence-like units that are expected by conventional machine translation (MT) systems for Spoken Language Translation. In this work, we propose a model for correcting the acoustic segmentation of ASR models for low-resource languages to improve performance on downstream tasks. We propose the use of subtitles as a proxy dataset for correcting ASR acoustic segmentation, creating synthetic acoustic utterances by modeling common error modes. We train a neural tagging model for correcting ASR acoustic segmentation and show that it improves downstream performance on MT and audio-document cross-language information retrieval (CLIR).

## introduction typical asr systems segment the input audio into utterances using purely acoustic information, i.e., pauses in speaking or other dips in the audio signal, which may not resemble the sentence-like units that are expected by conventional mt systems for spoken language translation (slt) @xcite . longer utterances may span multiple sentences, while shorter utterances may be sentence fragments containing only a few words (see figure 1 for examples). both can be problematic for downstream mt systems. in this work, we propose a model for correcting the acoustic segmentation of an asr model to improve performance on downstream tasks, focusing on the challenges inherent to slt pipelines for low-resource languages. while prior work has trained intermediate components to segment asr output into sentence-like units @xcite , these have primarily focused on highly resourced language pairs such as arabic and chinese. when the source language is low-resource, suitable training data may be very limited for asr and mt, and even nonexistent for segmentation. since typical low-resource language asr datasets crawled from the web do not have hand-annotated segments, we propose deriving proxy segmentation datasets from film and television subtitles. subtitles typically contain segment boundary information like sentence-final punctuation, and while they are not exact transcriptions, they are closer to transcribed speech than many other large text corpora. our proposed model takes as input a sequence of tokens and segmentation boundaries produced by the acoustic segmentation of the asr system and returns a corrected segmentation. while subtitles are often similar to speech transcripts, they lack an existing acoustic segmentation for our model to correct. to account for this, we generate synthetic acoustic segmentation by explicitly modeling two common error modes of asr acoustic segmentation: under-and over-segmentation. we evaluate the downstream mt performance in a larger slt pipeline, and show improvements in translation quality when using our segmentation model to correct the acoustic segmentation provided by asr. we also extrinsically evaluate our improved slt pipeline as part of a documentlevel cross-lingual information retrieval (clir) task, where we show that improvements in asr segmentation also lead to improved relevance of search results. we report results for nine translation settings: bulgarian (bg) to english, lithuanian (lt) to english, and farsi (fa) to english, and when using either phrase-based, statistical mt (smt) or one of two neural mt (nmt) models. we finally perform an ablation study to examine the effects of our synthetic acoustic boundaries and our over-and under-segmentation noise. this paper makes the following contributions. (i) we propose the use of subtitles as a proxy dataset for correcting asr acoustic segmentation and (ii) a method for adding synthetic acoustic utterance segmentations to a subtitle dataset, as well as (iii) a simple neural tagging model for correcting asr acoustic segmentation before use in an mt pipeline. (iv) finally, we show downstream performance increases on mt and document-level clir tasks, especially for more syntactically complex segments. ## related work segmentation in slt has been studied quite extensively in high-resource settings. early work used kernel-based svm models to predict sentence boundaries using language model probabilities along with prosodic features such as pause duration @xcite and part-of-speech features derived from a fixed window size @xcite . other work has modeled the problem using hidden markov models @xcite @xcite and conditional random fields @xcite . more recent segmentation work uses neural architectures, such as lstm @xcite and transformer models @xcite . these models benefit from the large training data available for high-resource languages. for example, the ted corpus @xcite for slt from english to german includes about 340 hours of welltranscribed data. to our knowledge, such datasets do not exist for the languages we are interested in. @xcite develop a segmentation model in our setting using subtitles; however, they do not take into account explicit modeling of segmentation errors and show only minimal and intermittent improvements in downstream tasks. recent work has increasingly focused on endto-end models of slt in a high-resource setting, since these systems reduce error propagation and latency when compared to cascaded approaches @xcite @xcite @xcite . in spite of these advantages, end-to-end systems have only very recently achieved competitive results due to the limited amount of parallel data for speech translation as compared to the data that is available to train asr systems and translation systems separately @xcite . ## problem definition we treat the asr acoustic segmentation problem as a sequence tagging problem @xcite . unlike a typical tagging problem, which aims to tag a single input sequence, our input is a pair of aligned sequences of n items, @xmath0, . . . , x n ] and γ = [γ 1 , . . . , γ n ] where x and γ are the asr tokens and acoustic segmentation respectively. the tokens x i belong to a finite vocabulary v, while the acoustic segmentation boundary tags are binary, i.e., γ @xmath1, 1}, where γ @xmath2. the goal is to predict a corrected segment boundary tag sequence @xmath3, . . . , y n ] ∈ {0, 1} n from x and γ. we do this by learning a probabilistic mapping from token/segmentation sequences to corrected segmentation p(•|x, γ; θ) : {0, 1} @xmath4, 1) where p is a neural tagging model with parameters θ. while γ are produced solely from acoustic cues, p can take advantage of both the acoustic information (via γ) as well as syntactic/semantic cues implicit in x. reference segmentation: ## generating training data from subtitles one of our primary contributions is a method for converting subtitle data into suitable training data for an asr segmentation correction model. the subtitle data contains speech-like utterances of dialogue between characters in film and television shows. for the purposes of this paper, we do not use information about speaker identity, only the text and information about segmentation. we obtain the ground truth output label segmentation y by segmenting the subtitle text on sentence final punctuation. 1 we remove the punctuation but keep the implied label sequence to obtain the input token sequence x and ground truth output label segmentation y. however, we do not have acoustic segmentation available for x, y pairs derived from subtitle data, which we will need as additional input if our model is to learn to correct acoustic segmentation provided by an asr component. we thus create a synthetic acoustic segmentation sequence γ as input by adding two types of noise to y. specifically, we imitate two common asr system errors, undersegmentation noise and over-segmentation, so that at test time the model can correct those errors. under-segmentation noise in the asr model, under-segmentation occurs when pauses between words are brief, and the resulting asr output is an utterance that could ideally be split into multiple sentence-like segments. we simulate this by adding 1 set of sentence final punctuation: { ( ) : -! ? . }. over-segmentation noise over-segmentation occurs in an asr model when a speaker takes a longer pause in the middle of what could be interpreted as a contiguous sentence-like utterance. over-segmentation noise is simulated by inserting random segment boundaries within an utterance. that is, with probability α we convert a non-boundary tag y @xmath5, while leaving all y @xmath6. ## model we employ a long short-term memory (lstm)based model architecture for this task @xcite . given an input sequence of asr tokens @xmath7, . . . , x n ] along with corresponding asr segmentation sequence γ = [γ 1 , . . . , γ n ], we first get an embedding representation e @xmath8: each output state h i is then passed through a linear projection layer with a logistic sigmoid to compute the probability of a segment boundary p(y @xmath9; θ). the log-likelihood of a corrected segmentation boundary sequence is log p(y|x, γ; θ) = n @xmath10; θ). we fit the parameters, θ, by approximately minimizing the negative log-likelihood on the training set d, l(θ) = -1 |d| (x,γ,y)∈d log p (y|γ, x; θ), using mini-batch stochastic gradient descent. ## ablation study we perform an ablation study on two components in our proposed model, (i) the use of acoustic segmentation boundary labels γ as input and (ii) training with a combination of over-and undersegmentation noise. we use the same training and ## conclusion we propose an asr segmentation correction model for improving slt pipelines. our model makes use of subtitles data as well as a simple model of acoustic segmentation error to train an improved asr segmentation model. we demonstrate downstream improvements on mt and clir tasks. in future work, we would like to find a better segmentation error model that works well in conjunction with smt systems in addition to nmt systems.