Adapting End-to-End Speech Recognition for Readable Subtitles (2020)

Abstract: Automatic speech recognition (ASR) systems are primarily evaluated on transcription accuracy. However, in some use cases such as subtitling, verbatim transcription would reduce output readability given limited screen size and reading time. Therefore, this work focuses on ASR with output compression, a task challenging for supervised approaches due to the scarcity of training data. We first investigate a cascaded system, where an unsupervised compression model is used to post-edit the transcribed speech. We then compare several methods of end-to-end speech recognition under output length constraints. The experiments show that with limited data far less than needed for training a model from scratch, we can adapt a Transformer-based ASR model to incorporate both transcription and compression capabilities. Furthermore, the best performance in terms of WER and ROUGE scores is achieved by explicitly modeling the length constraints within the end-to-end ASR system.

## introduction automatic speech recognition (asr) has become ubiquitous in human interaction with digital devices, such as keyboard voice inputs @xcite and virtual home assistants @xcite . while transcription accuracy is often the primary goal when designing asr systems, in some use cases the readability of outputs is crucial to user experience. a prominent example is subtitling for tv. in this case, the audience need to multitask, i.e. simultaneously watch video contents, listen to speech utterances, and read subtitles. to avoid a visual overload, not every spoken word needs to be displayed. meanwhile, the shortened subtitles must still retain the meaning of the spoken content. moreover, large deviations from the original utterance are also undesirable, as the disagreement with auditory input would create a distraction. to compress the subtitles, one straightforward approach is to post-process asr transcriptions. the task of sentence compression has been wellstudied @xcite @xcite . in extractive compression @xcite , only deletion operations are performed on the input. despite the simplicity, this approach tends to produce outputs that are less grammatical @xcite . on the other hand, abstractive compression @xcite @xcite involves more sophisticated input reformulation, such as word reordering and paraphrasing @xcite . for the task of compressing subtitles, however, the extent of rewriting must be controlled in order to retain consistency with spoken utterances. from a practical point of view, building a sentence compression system typically requires training corpora where the target sequences are summarized. for most languages and domains, there exists scarcely any resource suitable for supervised training. this low-resource condition is even more severe for audio inputs. to the best of our knowledge, currently there is no publicly available spoken language compression corpora. given the challenges outlined above, this work investigates asr with output compression. we test our approaches on german tv subtitles. the combination of this task and the use case is to the extent of our knowledge previously unexplored. the first contribution of this work is a comparison of cascaded and end-to-end approaches to generating compressed asr transcriptions, where the former consists of separate asr and compression modules, and the latter integrates transcription and compression. the experiments show that our sentence compression module trained in an unsupervised fashion tends to excessively paraphrase, whereas the end-to-end model can be better adapted to the task of interest. secondly, we show that, after fine-tuning on a small adaption corpus, an asr model can perform transcription and compression simultaneously. without being given explicit length constraints, the adapted model shows increased recognition accuracy on rare words as well as paraphrasing capabilities to produce shorter outputs. furthermore, by explicitly encoding the length constraints, we achieve further performance gains in addition to those brought by adaptation. ## task the task of creating readable subtitles for video contents has several unique properties. first, due to limited screen size and reading time, not every spoken word needs to be transcribed, especially when utterances are spoken fast. a full transcription could even hamper user experience due to poor readability. second, although output shortening is typically realized by deleting non-essential words, the output is not only deletion-based. a real-life example from the german tv program tagesschau 1 shown in within the framework of common nlp tasks, the task of generating readable subtitles combines asr and abstractive compression, while being subjected to the additional requirements as outlined above. spoken: befreit vom fraktionszwang soll das parlament wohl nach der sommerpause die ethisch schwierige frage debattieren. (freed from pressure from the coalition party, the parliamentary should debate the ethically difficult question after the summer break.) subtitle: ohne fraktionszwang soll das parlament wohl nach der sommerpause die schwierige frage debattieren. (without pressure from the coalition party, the parliamentary should debate the difficult question after the summer break.) ## conclusion in this work, we explored the task of compressing asr outputs to enhance subtitle readability. this task has several unique properties. first, the compression is not solely deletion-based. moreover, unnecessary paraphrasing must be limited to maintain a consistent user experience between hearing and reading. we first investigated cascading an asr module with a sentence compression model. due to the absence of supervised corpora, the compression model is trained in an unsupervised fashion. experiments showed that the outputs generated this way do not suit our task requirements because of unnecessary paraphrasing. we then adapted an end-toend asr model on a small corpus with compressed transcriptions. via adaptation, the model learned to both shorten its outputs and improve transcription quality. nevertheless, the given length constraints were not fully satisfied. lastly, by explicitly injecting length constraints via reverse positional encoding, we achieved further performance gain, while completely adhering to length constraints. a direction for future work is to incorporate more diverse measurements of output length as well as complexity. in this work, we measured length by the number of bpe-tokens. while this typically corresponds to the output length perceived visually, a more direct metric would be the number of characters. moreover, output complexity, such as the proportion of long words, is also important for readability and therefore worth exploring. in a broader scope, as an anonymous reviewer suggested, a way to alleviate the resource scarcity for end-to-end asr compression is to augment the training data with synthesized utterances from summarization corpora. we expect the augmentation to be complementary to our approaches in this work.