BERT Meets CTC : New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model (2022)

Abstract: This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC’s training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.

## introduction the field of natural language processing (nlp) has witnessed remarkable improvements in performance thanks to the advances in deep learningbased techniques @xcite @xcite @xcite . much of the recent progress in nlp lies in large-scale language models (lms) @xcite , which are pre-trained on a vast amount of text data to learn versatile linguistic knowledge @xcite . such pre-trained models have been shown to improve diverse nlp tasks, alleviating the heavy requirement of supervised training data. inspired by the great success in nlp, pre-trained lms have been actively adopted for speech processing tasks, including automatic speech recognition (asr) @xcite , spoken language understanding (slu) @xcite , and text-to-speech synthesis @xcite . this paper focuses on leveraging pre-trained lms for end-to-end asr (e2e-asr), which aims to model direct speech-to-text conversion @xcite @xcite . one of the challenges in e2e-asr is a huge discrepancy between input and output sequences; the input is a continuous acoustic signal with finegrained patterns, while the output is discrete linguistic symbols (e.g., words) with long-range dependencies. such an input-output gap makes it difficult for an e2e-asr model to extract semantic/morphosyntax information from speech, which is essential for generating proper text. we believe this limitation can be mitigated by taking advantage of the rich linguistic representations obtained from pre-trained lms. several attempts have been made to use pretrained lms indirectly for improving e2e-asr, such as n-best hypothesis rescoring @xcite @xcite @xcite and knowledge distillation @xcite @xcite . others have investigated directly unifying an e2e-asr model with a pre-trained lm, where the lm is fine-tuned to optimize asr in an end-to-end trainable framework @xcite @xcite @xcite . we explore a novel direction for adopting a pre-trained masked language model (mlm) for e2e-asr, based on connectionist temporal classification (ctc) @xcite . compared to other autoregressive approaches, such as rnn-transducer (rnn-t) @xcite and attention-based sequence-to-sequence @xcite , ctc's non-autoregressive formulation allows simple training and inference processes for realizing e2e-asr. however, the performance of ctc is often limited due to a conditional independence assumption between output tokens @xcite . in this work, we propose bert-ctc that adapts bert @xcite for ctc to mitigate the conditional independence assumption. bert-ctc conditions ctc outputs on context-aware bert embeddings, thereby incorporating explicit linguistic information into training/inference. the bert-conditional formulation enables a model to attend to the full contexts of the input and hypothesized output sequences via the self-attention mechanism, while maintaining the benefits of a simple training algorithm in ctc. during inference, bert-ctc combines a mask-predict algorithm with ctc decoding, which iteratively refines outputs with flexible length adjustment. the key contributions of this work are summarized as follows: ## background to understand how bert-ctc exploits bert for relaxing the conditional independence assumption in ctc, we start with a brief review of probabilistic formulations of conventional e2e-asr approaches, including ctc @xcite and rnn-t @xcite . definition of end-to-end asr let @xmath0, • • • , t ) be an input sequence of length 1 https://github.com/yosukehiguchi/espnet/tree/ bert-ctc t , and @xmath1, • • • , n ) be the corresponding output sequence of length n . here, o t is a d-dimensional acoustic feature at frame t, w n is an output token at position n, and v is a vocabulary. 2 in general, the output length is much shorter than the input length (i.e., n ≪ t ). the objective of asr is to find the most probable output sequence ŵ that corresponds to a given input sequence o: ## bert-ctc overview in fig. 1 , we compare our proposed e2e-asr model, bert-ctc, to ctc and rnn-t. bert-ctc leverages powerful representations from bert @xcite to make ctc training/inference explicitly conditioned on linguistic information (fig. 1 (a) vs. fig. 1(c )). we use bert as a feature extractor for a (masked) token sequence, whose parameters are frozen during training. bert-ctc can be similar to rnn-t in that audio and token representations are fused to estimate the distribution over alignments (fig. 1(b vs. fig. 1(c )). however, bert-ctc attends to the full contexts of the input and output sequences via the self-attention mechanism @xcite , which facilitates a model to learn inner/interdependencies within/between the sequences. bert-ctc is formulated by introducing a partially masked (or partially observed) sequence @xmath2, • • • , n ), which is obtained by replacing some tokens in an output sequence w with a special mask token @xcite . note that during inference, we apply masks to a hypothesized sequence ŵ to obtain a masked sequence. considering all possible w , the conditional probability p(w |o) is factorized as follows: the conditional probability p(w | w , o) is further factorized by using the ctc alignment as in eq. ( 15 ), we make two conditional independence assumptions. the first is that given w and o, w similar to ctc, the joint probability p(a|w, o) is factorized using the probabilistic chain rule as to obtain eq. ( 16 ), we make the same conditional independence assumption as in ctc. however, compared to eq. ( 4 ), eq. ( 16 ) is conditioned on an output sequence w , enabling a model to explicitly use linguistic information to estimate the distribution over alignments. this is somewhat similar to rnn-t (eq. ( 10 )), but is different in that bert-ctc attends to the whole context (w 1 , • • • , w n ). we discuss this advantage in §7.1. substituting eq. ( 16 ) into eq. ( 15 ), we model the product of p(a t |w, o) and p(w | w ) as eq. ( 15 ) in eq. ( 18 ), selfattn t (•) indicates the t-th output of stacked transformer self-attention layers @xcite , which consume the concatenated h ae (from eq. ( 6 )) and h bert . 4 in eq. ( 19 ), bert(•) embeds a masked sequence w into a sequence of ## additional related work end-to-end asr with mlm inspired by the great success in non-autoregressive neural machine translation, conditional masked language model (cmlm) @xcite has been adopted for e2e-asr. audio-cmlm (a-cmlm) @xcite has trained an e2e-asr model with an mlm objective @xcite , making token predictions conditioned on both the speech input and a partially masked target sequence. imputer @xcite and mask-ctc @xcite have introduced ctc to the cmlm-based modeling, where the mask-predict algorithm is used to refine a framelevel or token-level sequence predicted by ctc. our method of combining ctc and mlm is related to the above studies, but conceptually different in that bert-ctc aims to relax the conditional independence assumption used in ctc by leveraging an external pre-trained mlm (i.e., bert) as contextual embedding. ## experiments we used the espnet toolkit @xcite for all the experiments. all the implementations and recipes are made publicly available (see §1). ## results speech recognition figure 2 illustrates the correlation between bert-ctc results and the number of decoding iterations. when decoded with @xmath3, the model only uses speech input to predict a token sequence. by increasing k, the model beneficially exploited the bert knowledge for refining the output tokens. spoken language understanding ## analyses 7.1 ablation studies to validate the effectiveness of our model design for bert-ctc, we conduct ablation studies ( ## conclusion we proposed bert-ctc that leverages bert for relaxing the conditional independence assumption in ctc. bert-ctc uses bert as contextual embedding to explicitly condition ctc training/inference on linguistic information. experimental results showed that bert-ctc improved over conventional approaches. moreover, we confirmed that bert-ctc is applicable to end-to-end slu.