Efficient CTC Regularization via Coarse Labels for End-to-End Speech Translation (2023)

Abstract: For end-to-end speech translation, regularizing the encoder with the Connectionist Temporal Classification (CTC) objective using the source transcript or target translation as labels can greatly improve quality. However, CTC demands an extra prediction layer over the vocabulary space, bringing in non-negligible model parameters and computational overheads, although this layer becomes useless at inference. In this paper, we re-examine the need for genuine vocabulary labels for CTC for regularization and explore strategies to reduce the CTC label space, targeting improved efficiency without quality degradation. We propose coarse labeling for CTC (CoLaCTC), which merges vocabulary labels via simple heuristic rules, such as using truncation, division or modulo (MOD) operations. Despite its simplicity, our experiments on 4 source and 8 target languages show that CoLaCTC with MOD particularly can compress the label space aggressively to 256 and even further, gaining training efficiency (1.18× ∼ 1.77× speedup depending on the original vocabulary size) yet still delivering comparable or better performance than the CTC baseline. We also show that CoLaCTC successfully generalizes to CTC regularization regardless of using transcript or translation for labeling.

## introduction developing techniques to support the translation from a source-language audio to a target-language text directly, or end-to-end (e2e) speech translation (st), has attracted increasing attention recently due to its potential of reducing translation latency and avoiding error propagation @xcite . however, solving this task is non-trivial because of the speech-text modality gap: one word corresponds to a stochastic sequence of speech signals that vary greatly across speakers and over contexts, which increases the learning difficulty. recent progress on e2e st mainly focuses on bridging this gap through the encoder-decoder framework from diverse perspectives @xcite @xcite @xcite . ctc regularization is such an approach that facilitates the modeling of translation by aligning speech representations from the encoder with discrete labels dynamically via the lens of the connectionist temporal classification (ctc) objective @xcite . @xcite first examined the use of the source transcript as discrete labels, improving translation quality consistently across various st settings; @xcite further discovered that using the target translation as labels instead can also be surprisingly effective although speech-translation pairs arguably violates ctc's monotonicity prerequisite. nevertheless, these successes come at the cost of increased computational overheads and model parameters because ctc demands an extra prediction layer over its label space for probability estimation and this space is often huge -traditionally the source or target vocabulary size @xcite . we thus explore strategies to achieve the best of both worlds, i.e., improving the efficiency of ctc regularization without hurting its performance. we address this problem by reexamining the need for genuine vocabulary labels for ctc. in contrast to ctc-based generation @xcite , the prediction layer in ctc regularization of st is discarded at inference. in other words, sticking to genuine labels is computationally unnecessary. since the large label space of ctc is a crucial bottleneck hindering training efficiency, we explore ways of reducing it. we propose coarse labeling for ctc (colactc) that manipulates this space by merging vocabulary labels based on simple heuristic rules. concretely, we map the source or target vocabulary to a pseudo label space subject to some predefined size using simple op-method mapping # labels id sequence genuine labels f(z) = z v 0,1,2,3,4,5,6,7,8 colactc truncation f(z) = min(z, l-1) l 0,1,2,2,2,2,2,2,2 modulo f(z) = z % l l 0,1,2,0,1,2,0,1,2 division f(z) = ⌊z / (v/l)⌋ l 0,0,0,1,1,1,2,2,2 log-scaling f(z) = ⌊log(max(z,1)) * l/log(v)⌋ l 0,0,0,1,1,2,2,2,2 despite the label space being transformed, the generated coarse labels still maintain a strong correlation with their vocabulary counterparts, ensuring their informativeness for representation learning. we rigorously examined our method on the must-c @xcite and the multilingual tedx @xcite benchmarks, covering 4 source languages and 8 target languages. across diverse settings, colactc successfully achieves comparable or even better translation performance than the ctc baseline but with significantly improved training efficiency (up to 1.77× speedup depending on the original vocabulary size). our main contributions are summarized below: 1 solving e2e st requires techniques to mitigate the speech-text modality gap. one way is to develop 1 source code: https://github.com/bzhanggo/zero . in this study, we mainly focus on bilingual st using triplet data alone, and leave the exploration of how our method is compatible with other setups to future work. ctc was first proposed to handle the sequence mismatch problem between acoustic features and transcript tokens, and has been widely applied to asr @xcite and other tasks where the input sequence is longer than the output and their alignment is monotonic @xcite . recent studies also show promising results when applying ctc to non-monotonic tasks, specifically to nonautoregressive st and mt @xcite @xcite . all these methods treat the prediction layer in ctc as a generator, used to predict final outputs. by contrast, @xcite used ctc as an auxiliary objective to improve asr. in e2e st, @xcite , @xcite , @xcite and @xcite leveraged ctc to compress speech representations to bridge the modality gap; @xcite and @xcite explored ctc as a regularizer using source transcript as labels, showing encouraging performance although the prediction layer is not used at inference; @xcite further investigated e2e st without transcript, and found that ctc regularization with translation as labels also works. our study follows ctc regularization and extends it with colactc to address its training inefficiency issue. as far as we know, exploring coarse labels for ctc has never been investigated before, at least on st. the translation loss l mle aims at maximizing the likelihood of observed training instances. often, we decompose the likelihood token-wise in accordance with the autoregressive generation by contrast, the regularization term l ctc encourages the dynamic alignment of speech representations (x) with their corresponding discrete labels (z) through the ctc algorithm. ctc regularization also maximizes the likelihood. but different from the token-by-token formulation in eq. ( 2 ), ctc estimates the likelihood by marginalizing over all valid mappings between the input and output sequence @xcite ) previous studies have examined using either source transcript @xcite or target translation @xcite as labels for ctc regularization, but separately. in section 5.1, we will compare these two types of ctc regularization under the same setup. ## coarse labeling for ctc unfortunately, ctc regularization suffers from training inefficiency. despite ctc being efficiently addressed via dynamic programming, its prediction layer in eq. ( 5 ) is unavoidable. this layer introduces considerable computational overhead due to scaling linearly with the vocabulary size and also brings in large number of model parameters particularly when v ̸ = v mle . how to improve the efficiency, and save model parameters while retaining the performance is the focus of our study. we draw inspiration from the fact that the prediction layer in ctc regularization is not used for inference. thus, sticking to genuine vocabulary labels is unnecessary technically. if we could design pseudo labels in a reduced label space as alternatives to the genuine ones, that would address the inefficiency issue. following this intuition, we propose coarse labeling for ctc (colactc) which formulates the pseudo label generation process as a vocabulary mapping: one assumption behind such formulation is that the success of ctc regularization mainly comes from the inductive biases of the ctc algorithm rather than the genuineness of its labels foot_2 , which we verified empirically through experiments. eq. ( 6 ) merges a set of vocabulary labels into one label according to f (•). potential mappings are many, such as grouping semantically similar words or considering phonetic similarity. but these linguistically inspired approaches often lack freedom in manipulating the label space (l). instead, we adopt the following heuristic methods (as shown in modulo (mod) nevertheless, infrequent items might carry crucial content information. instead of naively collapsing them, mod merges diverse labels of varying frequencies based on a fixed interval division (div) different from items with varying frequencies, items of similar frequency often share similar linguistic properties. we explore this in div which merges labels of similar ranks uniformly log-scaling (log) one drawback of div is that the distribution of its coarse labels becomes badly skewed. to offset this problem, we further study a non-linear, log-scaled transformation, log note we intentionally use simple operations to keep the simplicity of colactc. all the above operations are trivial to implement. although labels generated by these operations become linguistically less meaningful, they still keep a strong correlation with their genuine vocabulary counterparts. we expect this correlation could ensure the informativeness of each coarse label and further facilitate the generalization of colactc to ctc regularization. we compare different operations via experiments. ## conclusion and future work in this paper, we have presented coarse labeling for ctc to address the training inefficiency issue of ctc regularization. the key idea behind co-lactc is to transform ctc labels from the vocabulary space to a specified and reduced coarse space. we adopt trivial mappings for this transformation, such as using the modulo operation. despite its simplicity, colactc successfully achieves the best of both worlds -improving the training efficiency for ctc regularization (up to 1.77× speedup) and retaining its quality benefits -and generalizes to different types of ctc regularization. note the training speedup scales as the vocabulary size increases. our analysis further shows that the genuineness of ctc labels matters less than their origin. in the future, we are interested in examining the complementarity of colactc with other advanced st modeling. we will study how our method performs in a multilingual and simultaneous setup as well as st settings with extra asr and/or mt data.