CTC -based Non-autoregressive Speech Translation (2023)

Abstract: Combining end-to-end speech translation (ST) and non-autoregressive (NAR) generation is promising in language and speech processing for their advantages of less error propagation and low latency. In this paper, we investigate the potential of connectionist temporal classification (CTC) for non-autoregressive speech translation (NAST).In particular, we develop a model consisting of two encoders that are guided by CTC to predict the source and target texts, respectively. Introducing CTC into NAST on both language sides has obvious challenges: 1) the conditional independent generation somewhat breaks the interdependency among tokens, and 2) the monotonic alignment assumption in standard CTC does not hold in translation tasks. In response, we develop a prediction-aware encoding approach and a cross-layer attention approach to address these issues. We also use curriculum learning to improve convergence of training. Experiments on the MuST-C ST benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67 \times , which is comparable to the autoregressive counterpart and even outperforms the previous best result of 0.9 BLEU points.

## introduction end-to-end speech translation (e2e st) has attracted unprecedented attention and achieved dramatic development in recent years @xcite @xcite @xcite @xcite . stand-alone modeling reduces the inference latency by almost half compared to cascaded systems, where the automatic speech recognition (asr) model and the machine translation (mt) model run serially. this helps the application in real scenarios, especially with limited computational resources. however, this advantage only holds in the context of autoregressive (ar) decoding, where each token is generated depending on the previously predicted results. non-autoregressive (nar) generation @xcite , the recently popular decoding method in asr and mt, makes the inference process fast by predicting the output sequence in parallel, resulting in the e2e st no longer being superior in terms of inference speed-up. a natural question arises: can we build a powerful non-autoregressive speech translation (nast) model? the nar results in the latest literature are still inferior to the ar counterparts with a large gap of about 2 ∼ 3 bleu points, even with the iterative refinement process @xcite . in this work, we aim to develop a promising nast model for comparable performance to the ar model without complex decoding. we resort to the connectionist temporal classification (ctc, @xcite because of its great success in asr and mt and the convenience of variable length prediction. ctc is well suited for speech-to-text modeling, where the input sequence is longer than the output. recent studies show that ctc-based nar models achieve comparable or even better performance than their ar counterparts, providing insight into the design of the powerful ctc-nast model. our ctc-nast model is decoder-free and consists of two stacked encoders: an acoustic encoder and a textual encoder. they are guided by ctc to predict transcription and translation, respectively @xcite . then, we carry out a careful and systematic inspection of the underlying issues and address the challenges of ctc-nast. in particular, even with the above efforts, nast is still a difficult task that suffers from heavy modeling burdens. a curriculum learning strategy that guides the training in an easy-to-hard way is significant for better convergence. we replace part of the incorrect prediction with ground truth in pae to prompt the generation of the whole sequence. in this way, the model relieves the ctc learning burden by observing almost the whole sequence in the early stages, while only a few tokens are replaced as ctc performance improves, ensuring consistency between training and inference. our ctc-nast model is simple, completely parallel, and works well for both similar and distant language pairs. the proposed methods yield a remarkable gain of 3.0 bleu points on must-c en-de, achieving an average bleu score of 29.5 with an inference speed-up of 5.67×, and even outperforming the best previous ar results by 0.9 bleu points. we also report competitive results on the more challenging must-c en-ja and fisher-callhome corpus. ctc has the same conditional independence property as nar generation, where the probability of the path π is the product of the probability p (π t |h t ) at each time step t: ## ctc-nast among many well-established nar designs for asr or mt models, ctc is particularly suitable for st modeling because the input length is remarkably longer than its output. in this section, we present ctc-nast in detail. we first describe the base architecture, then identify and address three underlying challenges. see figure 1 for an overview of our system. ## extension on ar model now a natural question arises: can our method proposed for the nar model be used to improve the ar model? our method produces better encoder representations for ctc prediction, but there is no evidence to demonstrate that the optimization of the ctc and the cross-entropy in the decoder are completely consistent. excessive optimization of the encoder may interfere with the learning of the decoder. to answer it, we adopt these techniques to the encoder-decoder counterpart (called ctc-aug st), to investigate the effects of different architectures. and the training loss is formulated as: ## experiments we evaluate our method on the must-c and fisher-callhome benchmarks. details about the datasets and model settings are described in appendix a. ## conclusion aiming to combine e2e st and nar generation, we propose ctc-nast, which consists of only two ctc-guided encoders for source and target text prediction, respectively. we identify and address several challenges of ctc-nast: conditional independence assumption, monotonic assumption, and poor convergence. in this way, our ctc-nast model outperforms the previous best ar models by 0.9 bleu points. we believe that we are the first to present a nast model that achieves comparable or better performance than strong ar counterparts.